[kubernetes]-k8s通过psp限制nvidia-plugin插件的使用

导语： k8s通过psp限制nvidia-plugin插件的使用。刚开始接触psp 记录一下 后续投入生产测试了再完善。

通过apiserver开启psp 静态pod会自动更新

# PSP(Pod Security Policy) 在默认情况下并不会开启。通过将PodSecurityPolicy关键词添加到 --enbale-admission-plugins 配置数组后，可以开启PSP权限认证功能。
# /etc/kubernetes/manifests/kube-apiserver.yaml   在NodeRestriction后添加PodSecurityPolicy
- --enable-admission-plugins=NodeRestriction,PodSecurityPolicy

直接创建容器测试

lung.yaml

apiVersion: apps/v1
kind: Deployment
metadata:name: lunglabels:k8s-app: lungk8s-med-type: biz-internel
spec:strategy:type: Recreatereplicas: 1selector:matchLabels:k8s-app: lungtemplate:metadata:labels:k8s-app: lungspec:
#      runtimeClassName: nvidia
#      hostPID: truecontainers:- name: lungimage: nvidia/cuda:11.3.0-base-ubi8command: ["sh","-c","tail -f /dev/null "]#command: ["sh","-c","for i in `ls /srv/conf-drwise220531`;do rm -rf /root/lung/$i/conf && ln -s  /srv/conf-drwise220531/$i/conf /root/lung/$i/  ;done  && rm -rf /root/lung/Release/path.conf /root/lung/path.conf  && ln -s /srv/conf-drwise220531/Release/path.conf /root/lung/Release/ && ln -s /root/lung/Release/path.conf  /root/lung/ &&  sh /root/aiclassifier/startup.sh &&  sh /root/lung/startup.sh "]
#        securityContext:
#          privileged: trueenv:- name: NVIDIA_DRIVER_CAPABILITIESvalue: compute,utility,video,graphics,display- name: NVIDIA_VISIBLE_DEVICESvalue: allvolumeMounts:- mountPath: /dev/shmname: dshmvolumes:- name: dshmemptyDir:medium: MemorysizeLimit: 1Gi
#deepwise-operator
#      serviceAccountName: deepwise-operator

# 创建deployment测试  发下会有psp的问题
# 注意：开启PodSecurityPolicy功能后，即使没有使用任何安全策略，都会使得创建pods（包括调度任务重新创建pods）失败kubectl apply -f lung -n deepwise 

创建对应的资源限制策略

4.nvidia-plugin.yaml

# 显卡驱动的限制
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:name: psp-nvidia
spec:privileged: falsefsGroup:rule: RunAsAnyrunAsUser:rule: RunAsAnyseLinux:rule: RunAsAnysupplementalGroups:rule: RunAsAnyvolumes:- "*"hostPID: falsehostIPC: falsehostNetwork: false---
apiVersion: v1
kind: ServiceAccount
metadata:namespace: kube-systemname: nvidiaoperator---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: psp-permissive-nvidianamespace: kube-system
rules:- apiGroups:- extensionsresources:- podsecuritypoliciesresourceNames:- psp-nvidiaverbs:- use
---apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:name: psp-permissive-nvidianamespace: kube-system
roleRef:apiGroup: rbac.authorization.k8s.iokind: Rolename: psp-permissive-nvidia
subjects:- kind: ServiceAccountname: nvidiaoperatornamespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:name: nvidia-device-plugin-daemonsetnamespace: kube-system
spec:selector:matchLabels:name: nvidia-device-plugin-dsupdateStrategy:type: RollingUpdatetemplate:metadata:# This annotation is deprecated. Kept here for backward compatibility# See https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/annotations:scheduler.alpha.kubernetes.io/critical-pod: ""labels:name: nvidia-device-plugin-dsspec:runtimeClassName: nvidiatolerations:# This toleration is deprecated. Kept here for backward compatibility# See https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/- key: CriticalAddonsOnlyoperator: Exists- key: nvidia.com/gpuoperator: Existseffect: NoSchedule# Mark this pod as a critical add-on; when enabled, the critical add-on# scheduler reserves resources for critical add-on pods so that they can# be rescheduled after a failure.# See https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/priorityClassName: "system-node-critical"containers:- image: harbor.deepwise.com/terra-k8s/k8s-device-plugin:v0.10.0name: nvidia-device-plugin-ctrargs: ["--fail-on-init-error=false"]securityContext:allowPrivilegeEscalation: falsecapabilities:drop: ["ALL"]volumeMounts:- name: device-pluginmountPath: /var/lib/kubelet/device-pluginsvolumes:- name: device-pluginhostPath:path: /var/lib/kubelet/device-pluginsserviceAccountName: nvidiaoperator

5.nvida-psp.yaml

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:name: psp-nvidia
spec:privileged: falsefsGroup:rule: RunAsAnyrunAsUser:rule: RunAsAnyseLinux:rule: RunAsAnysupplementalGroups:rule: RunAsAnyvolumes:- "*"hostPID: falsehostIPC: falsehostNetwork: false---
apiVersion: v1
kind: ServiceAccount
metadata:namespace: kube-systemname: nvidiaoperator---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: psp-permissive-nvidianamespace: kube-system
rules:- apiGroups:- extensionsresources:- podsecuritypoliciesresourceNames:- psp-nvidiaverbs:- use
---apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:name: psp-permissive-nvidianamespace: kube-system
roleRef:apiGroup: rbac.authorization.k8s.iokind: Rolename: psp-permissive-nvidia
subjects:- kind: ServiceAccountname: nvidiaoperatornamespace: kube-system

6.deepwise-psp.yaml

# 用户的限制
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:name: psp-deepwise
spec:privileged: falsefsGroup:rule: RunAsAnyrunAsUser:rule: RunAsAnyseLinux:rule: RunAsAnysupplementalGroups:rule: RunAsAnyvolumes:- "*"hostPID: falsehostIPC: falsehostNetwork: false---
apiVersion: v1
kind: ServiceAccount
metadata:namespace: deepwisename: deepwise-operator---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: psp-permissive-deepwisenamespace: deepwise
rules:- apiGroups:- extensionsresources:- podsecuritypoliciesresourceNames:- psp-deepwiseverbs:- use
---apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:name: psp-permissive-deepwisenamespace: deepwise
roleRef:apiGroup: rbac.authorization.k8s.iokind: Rolename: psp-permissive-deepwise
subjects:- kind: ServiceAccountname: deepwise-operatornamespace: deepwise

7.runtimeclass.yaml

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:name: "nvidia"
handler: "nvidia"

如果是docker运行时，handler需要调整为docker。使用containerd则不需要调整。参考https://opni.io/setup/gpu/

重新创建lung的deployment

# 加上runtimeClassName: nvidia
# 加上serviceAccountName: deepwise-operator
apiVersion: apps/v1
kind: Deployment
metadata:name: lunglabels:k8s-app: lungk8s-med-type: biz-internel
spec:strategy:type: Recreatereplicas: 1selector:matchLabels:k8s-app: lungtemplate:metadata:labels:k8s-app: lungspec:runtimeClassName: nvidia
#      hostPID: truecontainers:- name: lungimage: nvidia/cuda:11.3.0-base-ubi8command: ["sh","-c","tail -f /dev/null "]#command: ["sh","-c","for i in `ls /srv/conf-drwise220531`;do rm -rf /root/lung/$i/conf && ln -s  /srv/conf-drwise220531/$i/conf /root/lung/$i/  ;done  && rm -rf /root/lung/Release/path.conf /root/lung/path.conf  && ln -s /srv/conf-drwise220531/Release/path.conf /root/lung/Release/ && ln -s /root/lung/Release/path.conf  /root/lung/ &&  sh /root/aiclassifier/startup.sh &&  sh /root/lung/startup.sh "]
#        securityContext:
#          privileged: trueenv:- name: NVIDIA_DRIVER_CAPABILITIESvalue: compute,utility,video,graphics,display- name: NVIDIA_VISIBLE_DEVICESvalue: allvolumeMounts:- mountPath: /dev/shmname: dshmvolumes:- name: dshmemptyDir:medium: MemorysizeLimit: 1Gi
#deepwise-operatorserviceAccountName: deepwise-operator

参考文档

https://kubernetes.io/zh-cn/docs/reference/access-authn-authz/admission-controllers/

https://blog.csdn.net/tushanpeipei/article/details/121940757

https://blog.csdn.net/weixin_45081220/article/details/125407608