k8s 1.28.2 集群部署 NFS server 和 NFS Subdir External Provisioner

前言

NFS Subdir External Provisioner 可以使用现有的 NFS 服务器动态创建 pv 和 pvc

• nfs-subdir-external-provisioner

部署 NFS server

镜像准备

这块可以看我之前的博客，这里就不啰嗦了

• 使用 docker 的方式部署 NFS server 提供文件共享能力

节点打标签

采用 hostpath 的方式来持久化 NFS 的共享目录，需要绑定节点不让 NFS 飘移

k label node 192.168.22.124 nfs-server=true

启动 NFS server

• 这里记录两个问题
  • NFS 的配置文件，根目录或者说第一个共享目录，需要加上 fsid=0 ，然后挂载的时候直接使用 / ，如果不加 fsid=0，挂载会报错找不到文件或目录，细节什么的，可以看一下官方的手册：exports
  • 因为需要本地宿主机挂载 NFS 共享目录到 kubelet 的目录下面，宿主机就没办法使用 svc 的方式来挂载，除非本地 DNS 服务器包含了 k8s 集群内的 DNS，我这边就暂时使用指定的 clusterIP 地址来创建 svc，集群内直接使用 svc 的 ip 地址来挂载 NFS
    • 关于 clusterip 的 ip 范围，需要看 apiserver 的 --service-cluster-ip-range 参数，一般都是 10.96.0.0/12，可用的范围在 10.96.0.0 到 10.111.255.255 之间，找一个集群内不存在的 ip 来用就行
    • Service ClusterIP 分配
    • exports 里面要把 node 节点的 ip 网段，svc 的网段和 pod 的网段都写进去，如果嫌烦，也可以直接写 * ，只要不是对外暴露的，问题不是很大

---
apiVersion: v1
data:exports: |/nfs-share-data 192.168.22.0/24(rw,fsid=0,sync,no_subtree_check,no_auth_nlm,insecure,no_root_squash)/nfs-share-data 10.96.0.0/12(rw,fsid=0,sync,no_subtree_check,no_auth_nlm,insecure,no_root_squash)/nfs-share-data 172.22.0.0/16(rw,fsid=0,sync,no_subtree_check,no_auth_nlm,insecure,no_root_squash)
kind: ConfigMap
metadata:name: nfs-server-cmnamespace: storage
---
apiVersion: v1
kind: Service
metadata:labels:app.kubernetes.io/name: nfs-servername: nfs-server-svcnamespace: storage
spec:clusterIP: 10.111.111.111ports:- name: tcpport: 2049targetPort: tcpselector:app.kubernetes.io/name: nfs-servertype: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:labels:app.kubernetes.io/name: nfs-servername: nfs-servernamespace: storage
spec:replicas: 1selector:matchLabels:app.kubernetes.io/name: nfs-serverstrategy:type: RollingUpdatetemplate:metadata:labels:app.kubernetes.io/name: nfs-serverspec:affinity:nodeAffinity:requiredDuringSchedulingIgnoredDuringExecution:nodeSelectorTerms:- matchExpressions:- key: nfs-serveroperator: Invalues:- "true"containers:- env:- name: SHARED_DIRECTORYvalue: /nfs-share-dataimage: nfs-server-2.6.4:alpine-3.20imagePullPolicy: IfNotPresentname: nfs-serverports:- containerPort: 2049name: tcpprotocol: TCPresources:limits:cpu: 1000mmemory: 1024Mirequests:cpu: 100mmemory: 100MisecurityContext:capabilities:add:- SYS_ADMINvolumeMounts:- mountPath: /nfs-share-dataname: nfs-share-data- mountPath: /etc/exportsname: nfs-configsubPath: exportsvolumes:- hostPath:path: /approot/k8s_data/nfs-share-datatype: DirectoryOrCreatename: nfs-share-data- configMap:name: nfs-server-cmname: nfs-config

创建 pv 验证

---
apiVersion: v1
kind: PersistentVolume
metadata:name: nfs-pv
spec:capacity:storage: 1GiaccessModes:- ReadWriteManynfs:server: 10.111.111.111path: "/"

创建 pvc

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: nfs-pvc
spec:accessModes:- ReadWriteManyresources:requests:storage: 1Gi

创建 pod 挂载验证

---
apiVersion: v1
kind: Pod
metadata:name: nfs-client
spec:containers:- name: appimage: m.daocloud.io/busybox:1.37command: ["sh", "-c", "while true; do sleep 3600; done"]volumeMounts:- name: nfs-storagemountPath: /mnt/nfsvolumes:- name: nfs-storagepersistentVolumeClaim:claimName: nfs-pvc

如果 pod 启动有类似如下的报错，可以在 k8s 节点上安装一下 nfs-utils

  Warning  FailedMount  1s (x7 over 33s)  kubelet            MountVolume.SetUp failed for volume "nfs-pv" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs nfs-server-svc.storage.svc.cluster.local:/nfs-share-data /var/lib/kubelet/pods/9e7abc6f-573c-4c3f-b023-cdceee95722a/volumes/kubernetes.io~nfs/nfs-pv
Output: mount: /var/lib/kubelet/pods/9e7abc6f-573c-4c3f-b023-cdceee95722a/volumes/kubernetes.io~nfs/nfs-pv: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program.

部署 NFS Subdir External Provisioner

官方也有 helm 的文档，需要用 helm 的，可以直接看官方的：NFS Subdirectory External Provisioner Helm Chart

我这边采用 yaml 编排来部署

---
apiVersion: v1
kind: ServiceAccount
metadata:labels:app: nfs-subdir-external-provisionername: nfs-subdir-external-provisioner-sanamespace: storage
---
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:labels:app: nfs-subdir-external-provisionername: nfs-client
parameters:archiveOnDelete: "true"pathPattern: /
provisioner: cluster.local/nfs-subdir-external-provisioner
reclaimPolicy: Retain
volumeBindingMode: Immediate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:labels:app: nfs-subdir-external-provisionername: nfs-subdir-external-provisioner-runner
rules:
- apiGroups:- ""resources:- nodesverbs:- get- list- watch
- apiGroups:- ""resources:- persistentvolumesverbs:- get- list- watch- create- delete
- apiGroups:- ""resources:- persistentvolumeclaimsverbs:- get- list- watch- update
- apiGroups:- storage.k8s.ioresources:- storageclassesverbs:- get- list- watch
- apiGroups:- ""resources:- eventsverbs:- create- update- patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:labels:app: nfs-subdir-external-provisionername: run-nfs-subdir-external-provisioner
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: nfs-subdir-external-provisioner-runner
subjects:
- kind: ServiceAccountname: nfs-subdir-external-provisioner-sanamespace: storage
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:labels:app: nfs-subdir-external-provisionername: leader-locking-nfs-subdir-external-provisionernamespace: storage
rules:
- apiGroups:- ""resources:- endpointsverbs:- get- list- watch- create- update- patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:labels:app: nfs-subdir-external-provisionername: leader-locking-nfs-subdir-external-provisionernamespace: storage
roleRef:apiGroup: rbac.authorization.k8s.iokind: Rolename: leader-locking-nfs-subdir-external-provisioner
subjects:
- kind: ServiceAccountname: nfs-subdir-external-provisioner-sanamespace: storage
---
apiVersion: apps/v1
kind: Deployment
metadata:labels:app: nfs-subdir-external-provisionername: nfs-subdir-external-provisionernamespace: storage
spec:replicas: 1selector:matchLabels:app: nfs-subdir-external-provisionerstrategy:type: Recreatetemplate:metadata:labels:app: nfs-subdir-external-provisionerspec:containers:- env:- name: PROVISIONER_NAMEvalue: cluster.local/nfs-subdir-external-provisioner- name: NFS_SERVERvalue: 10.111.111.111- name: NFS_PATHvalue: /image: docker.m.daocloud.io/registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2imagePullPolicy: IfNotPresentname: nfs-subdir-external-provisionervolumeMounts:- mountPath: /persistentvolumesname: nfs-subdir-external-provisioner-rootserviceAccountName: nfs-subdir-external-provisioner-savolumes:- name: nfs-subdir-external-provisioner-rootnfs:path: /server: 10.111.111.111

创建 pod 验证

提前创建 pvc 的方式

创建 pvc

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:name: test-claim
spec:storageClassName: nfs-clientaccessModes:- ReadWriteManyresources:requests:storage: 1Mi

创建 pod

由于 NFS 这块是直接走的共享目录的根目录，会比较乱，因此 pod 增加了变量，再通过 volumeMounts.subPathExpr 将共享数据存到 pod 名字的目录下

---
kind: Pod
apiVersion: v1
metadata:name: test-pod
spec:containers:- name: test-podimage: m.daocloud.io/busybox:1.37command:- "/bin/sh"args:- "-c"- "touch /mnt/hello && exit 0 || exit 1"env:- name: POD_NAMEvalueFrom:fieldRef:apiVersion: v1fieldPath: metadata.namevolumeMounts:- name: nfs-pvcmountPath: "/mnt"subPathExpr: $(POD_NAME)restartPolicy: "Never"volumes:- name: nfs-pvcpersistentVolumeClaim:claimName: test-claim

可以通过之前创建的 nfs-client 来验证是否创建了 hello 这个文件

kubectl exec -it nfs-client -- ls /mnt/nfs/test-pod/

使用 volumeClaimTemplates 的方式

---
apiVersion: apps/v1
kind: StatefulSet
metadata:name: test-sts
spec:replicas: 1selector:matchLabels:app: test-ststemplate:metadata:labels:app: test-stsspec:containers:- name: test-stsimage: m.daocloud.io/busybox:1.37command:- "/bin/sh"args:- "-c"- "touch /mnt/SUCCESS && exit 0 || exit 1"env:- name: POD_NAMEvalueFrom:fieldRef:apiVersion: v1fieldPath: metadata.namevolumeMounts:- name: nfs-sts-pvcmountPath: "/mnt"subPathExpr: $(POD_NAME)restartPolicy: "Always"volumeClaimTemplates:- metadata:name: nfs-sts-pvcspec:storageClassName: nfs-clientaccessModes:- ReadWriteOnceresources:requests:storage: 10Gi

同样可以通过之前创建的 nfs-client 来验证是否创建了 SUCCESS 这个文件

kubectl exec -it nfs-client -- ls /mnt/nfs/test-sts-0