K8s:开源安全平台 kubescape 实现 Pod 的安全合规检查/镜像漏洞扫描

写在前面

---

• 生产环境中的 k8s 集群安全不可忽略，即使是内网环境
• 容器化的应用部署虽然本质上没有变化，始终是机器上的一个进程
• 但是提高了安全问题的处理的复杂性
• 分享一个开源的 k8s 集群安全合规检查/漏洞扫描 工具 kubescape
• 博文内容涉及：
  • kubescape 简介介绍
  • kubescape 命令行工具安装，扫描运行的集群
  • kubescape 在集群下安装，通过 kubescape Clound 可视化查看扫描信息
• 理解不足小伙伴帮忙指正
• 需要有科学上网环境

对每个人而言，真正的职责只有一个：找到自我。然后在心中坚守其一生，全心全意，永不停息。所有其它的路都是不完整的，是人的逃避方式，是对大众理想的懦弱回归，是随波逐流，是对内心的恐惧 ——赫尔曼·黑塞《德米安》

---

简单介绍

k8s 安全问题不可忽略

镜像的不可变性使我们能够方便地部署、测试和发布镜像到其他环境，这是一个很大的优势，但也带来了潜在的风险：镜像及其依赖在过时或者被弃用时，无法自动更新或打新的安全补丁，尤其一些基础镜像，重新做镜像，需要依赖一些厂商，或者开源项目 team 来维护。

RedHat 在 2022 年 Kubernetes 安全报告中对 300 多名 DevOps、工程和安全专业人士进行了调查，发现对 容器安全威胁的担忧以及对容器安全投资的缺乏 是 31% 的受访者对容器策略最普遍的担忧。

支持这些担忧的是 93% 的受访者在过去 12 个月内在其 Kubernetes 环境中经历过至少一次安全事件，这些事件有时会导致收入或客户流失。超过一半的受访者 (55%) 在过去一年中还因为安全问题而不得不推迟应用程序的推出。

尽管媒体广泛关注网络攻击，但该报告强调，实际上是错误配置让 IT 专业人员彻夜难眠。Kubernetes 是高度可定制的，具有可以影响应用程序安全状况的各种配置选项。因此，受访者最担心的是容器和 Kubernetes 环境中的错误配置导致的风险暴露 (46%) —— 几乎是对攻击的担忧程度 (16%) 的三倍。尽可能自动化配置管理有助于缓解这些问题，因此安全工具 —— 而不是人类 —— 提供了帮助开发人员和 DevOps 团队更安全地配置容器和 Kubernetes 的护栏。

以上内容来自：Redhat Blog(The State of Kubernetes Security in 2022)

Kubescape 简单介绍

Kubescape 是一个开源的 Kubernetes 安全平台。它的功能包括 风险分析、安全合规性 和 错误配置扫描。针对 DevSecOps 从业者或平台工程师，提供易于使用的 CLI 界面、灵活的输出格式和自动扫描功能。同时对于小集群提供了免费的 在线 面板工具，它为 Kubernetes 用户和管理员节省了宝贵的时间、精力和资源。

Kubescape 可以扫描运行的集群、静态 YAML 文件和 本地 Helm Charts。它根据多个框架（包括 NSA-CISA、MITRE ATT&CK®和CIS Benchmark）检测错误配置。

Kubescape 由 ARMO 创建，是一个 Cloud Native Computing Foundation (CNCF) 沙盒项目。

Kubescape 如果小伙伴觉得重的话， kubectl 有一个类似的插件，个人十分推荐，叫做 kube-score ,很轻量，也可以做一些简单的合规性扫描。但是只有扫描合规性的提示，没有规范出处。

Kubescape 安装

当前的集群信息

┌──[[e-mail]]-[~]
└─$kubectl get  nodes
NAME                          STATUS   ROLES           AGE   VERSION
vms100.liruilongs.github.io   Ready    control-plane   8d    v1.25.1
vms101.liruilongs.github.io   Ready    control-plane   8d    v1.25.1
vms102.liruilongs.github.io   Ready    control-plane   8d    v1.25.1
vms103.liruilongs.github.io   Ready    <none>          8d    v1.25.1
vms105.liruilongs.github.io   Ready    <none>          8d    v1.25.1
vms106.liruilongs.github.io   Ready    <none>          8d    v1.25.1
vms107.liruilongs.github.io   Ready    <none>          8d    v1.25.1
vms108.liruilongs.github.io   Ready    <none>          8d    v1.25.1
┌──[[e-mail]]-[~]
└─$

集群节点信息

┌──[[e-mail]]-[~]
└─$hostnamectlStatic hostname: vms100.liruilongs.github.ioIcon name: computer-vmChassis: vmMachine ID: e93ae3f6cb354f3ba509eeb73568087eBoot ID: a1150b6d97dc4afbb81dae58f131a487Virtualization: vmwareOperating System: CentOS Linux 7 (Core)CPE OS Name: cpe:/o:centos:centos:7Kernel: Linux 5.4.230-1.el7.elrepo.x86_64Architecture: x86-64
┌──[[e-mail]]-[~]
└─$

Kubescape 命令行工具安装

Kubescape CLI 安装

通过下面的方式自动安装

curl -s https://raw.githubusercontent.com/kubescape/kubescape/master/install.sh | /bin/bash

如果没有科学上网，找一台可以访问的集群，下载 install.sh 文件，按照下面方式修改，获取 curl 命令，自行下载。

┌──[[e-mail]]-[~/ansible/kubescape]
└─$vim install.sh
.......
# curl --progress-bar -L $DOWNLOAD_URL -o $OUTPUTecho "curl --progress-bar -L $DOWNLOAD_URL -o $OUTPUT"
exit 1
...

运行 shell，获取下载命令，找有网的机器下载

┌──[[e-mail]]-[~/ansible/kubescape]
└─$sh install.sh
Installing Kubescape...curl --progress-bar -L https://github.com/kubescape/kubescape/releases/latest/download/kubescape-ubuntu-latest -o /root/.kubescape/kubescape

上传到指定位置

┌──[[e-mail]]-[~/ansible/kubescape]
└─$ls
install.sh  kubescape-ubuntu-latest
┌──[[e-mail]]-[~/ansible/kubescape]
└─$mv kubescape-ubuntu-latest  /root/.kubescape/kubescape
mv：是否覆盖"/root/.kubescape/kubescape"？ y
┌──[[e-mail]]-[~/ansible/kubescape]
└─$

修改脚本，再次运行。

┌──[[e-mail]]-[~/ansible/kubescape]
└─$vim install.sh
# exit 1
┌──[[e-mail]]-[~/ansible/kubescape]
└─$sh install.sh
Installing Kubescape...curl --progress-bar -L https://github.com/kubescape/kubescape/releases/latest/download/kubescape-ubuntu-latest -o /root/.kubescape/kubescapeFinished Installation.Your current version is: v2.0.183 [git enabled in build: true]Usage: $ kubescape scan --enable-host-scan

到这里 ，kubescape 命令行工具即安装成功，扫描当前运行的集群可以运行如下命令。

┌──[[e-mail]]-[~/ansible]
└─$kubescape scan --enable-host-scan   --format html --output results.html  --verbose
[info] Kubescape scanner starting
[info] Installing host scanner
[info] Downloading/Loading policy definitions
[success] Downloaded/Loaded policy
[info] Accessing Kubernetes objects
[success] Accessed to Kubernetes objects
[info] Requesting images vulnerabilities results
[success] Requested images vulnerabilities results
[info] Requesting Host scanner data
[info] Host scanner version : v1.0.39
◑[error] failed to get data. path: /controlPlaneInfo; podName: host-scanner-xshr6; error: the server could not find the requested resource (get pods http:host-scanner-xshr6:7888)
◒[error] failed to get data. path: /controlPlaneInfo; podName: host-scanner-4tgnp; error: the server could not find the requested resource (get pods http:host-scanner-4tgnp:7888)
.......
[success] Done scanning. Cluster: kubernetes-admin-kubernetes^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^################################################################################
ApiVersion: hostdata.kubescape.cloud/v1beta0
Kind: KubeletInfo
Name: vms105.liruilongs.github.ioControls: 21 (Failed: 6, Excluded: 0)+----------+--------------------------------+------------------------------------+------------------------+
| SEVERITY |          CONTROL NAME          |                DOCS                | ASSISTANT REMEDIATION  |
+----------+--------------------------------+------------------------------------+------------------------+
| High     | CIS-4.1.7 Ensure that the      | https://hub.armosec.io/docs/c-0168 |                        |
|          | certificate authorities file   |                                    |                        |
|          | permissions are set to 600 or  |                                    |                        |
|          | more restrictive               |                                    |                        |
+          +--------------------------------+------------------------------------+------------------------+
|          | CIS-4.1.9 If the kubelet       | https://hub.armosec.io/docs/c-0170 |                        |
|          | config.yaml configuration      |                                    |                        |
|          | file is being used validate    |                                    |                        |
|          | permissions set to 600 or more |                                    |                        |
|          | restrictive                    |                                    |                        |
+----------+--------------------------------+------------------------------------+------------------------+
........
+----------+--------------------------------+------------------------------------+------------------------+
| Low      | CIS-4.2.6 Ensure that the      | https://hub.armosec.io/docs/c-0177 | protectKernelDefaults  |
|          | --protect-kernel-defaults      |                                    |                        |
|          | argument is set to true        |                                    |                        |
+          +--------------------------------+------------------------------------+------------------------+
|          | CIS-4.2.7 Ensure that the      | https://hub.armosec.io/docs/c-0178 | makeIPTablesUtilChains |
|          | --make-iptables-util-chains    |                                    |                        |
|          | argument is set to true        |                                    |                        |
+----------+--------------------------------+------------------------------------+------------------------+################################################################################
ApiVersion: v1
Kind: Namespace
Name: kubescapeControls: 18 (Failed: 17, Excluded: 0)+----------+--------------------------------+------------------------------------+----------------------------------------------------------------+
| SEVERITY |          CONTROL NAME          |                DOCS                |                     ASSISTANT REMEDIATION                      |
+----------+--------------------------------+------------------------------------+----------------------------------------------------------------+
| High     | CIS-5.2.11 Minimize the        | https://hub.armosec.io/docs/c-0202 | metadata.labels[pod-security.kubernetes.io/enforce]=baseline   |
|          | admission of Windows           |                                    |                                                                |
|          | HostProcess Containers         |                                    |                                                                |
+          +--------------------------------+------------------------------------+                                                                +
|          | CIS-5.2.2 Minimize the         | https://hub.armosec.io/docs/c-0193 |                                                                |
|          | admission of privileged        |                                    |                                                                |
|          | containers                     |                                    |                                                                |
+----------+--------------------------------+------------------------------------+----------------------------------------------------------------+
| Medium   | CIS-5.2.1 Ensure that          | https://hub.armosec.io/docs/c-0192 | metadata.labels[pod-security.kubernetes.io/enforce]=YOUR_VALUE |
|          | the cluster has at least       |                                    |                                                                |
|          | one active policy control      |                                    |                                                                |
|          | mechanism in place             |                                    |                                                                |
+          +--------------------------------+------------------------------------+----------------------------------------------------------------+
.......
..........
################################################################################
ApiVersion: hostdata.kubescape.cloud/v1beta0
Kind: ControlPlaneInfo
Name: vms102.liruilongs.github.ioControls: 25 (Failed: 1, Excluded: 0)+----------+--------------------------------+------------------------------------+-----------------------+
| SEVERITY |          CONTROL NAME          |                DOCS                | ASSISTANT REMEDIATION |
+----------+--------------------------------+------------------------------------+-----------------------+
| High     | CIS-1.1.20 Ensure that the     | https://hub.armosec.io/docs/c-0111 |                       |
|          | Kubernetes PKI certificate     |                                    |                       |
|          | file permissions are set to    |                                    |                       |
|          | 600 or more restrictive        |                                    |                       |
+----------+--------------------------------+------------------------------------+-----------------------+################################################################################
ApiVersion: apps/v1
Kind: Deployment
Name: local-path-provisioner
Namespace: local-path-storageControls: 35 (Failed: 18, Excluded: 0)
...
........
......

输出的合规信息和漏洞信息。

异常问题

我在 kubescape 多次次扫描中，集群因部分节点因为端口问题，无法发生调度，节点上的 kubescape 对于的 pod 无法自行删除，如果上一次扫描的 pod 或则 ns 没有删除，那么下一次的扫描无法进行，之前创建的 pod 和 ns 状态一直为 Terminating, 解决办法需要 对命名空间进行彻底删除。

这里执行完命令会进入阻塞状态。

┌──[[e-mail]]-[~/ansible/kubevirt]
└─$kubectl delete ns kubescape-host-scanner
namespace "kubescape-host-scanner" deleted

运行脚本删除

┌──[[e-mail]]-[~/ansible/k8s_shell_secript]
└─$cat delete_namespace.sh
#!/bin/bashcoproc kubectl proxy --port=30990 &if [ $# -eq 0 ] ; thenecho "后面加上你所要删除的ns."exit 1
fikubectl get namespace $1 -o json > logging.json
sed  -i '/"finalizers"/{n;d}' logging.json
curl -k -H "Content-Type: application/json" -X PUT --data-binary @logging.json http://127.0.0.1:30990/api/v1/namespaces/${1}/finalizekill %1
┌──[[e-mail]]-[~/ansible/k8s_shell_secript]
└─$sh delete_namespace.sh kubescape-host-scanner

也可以离线运行Kubescape，时间网络原因，这里不做分享，有需要的小伙伴可以到 github 上的项目地址查看详细信息。

一些其他的用法

扫描正在运行的 Kubernetes 集群：

kubescape scan --enable-host-scan  --verbose

使用替代的 kubeconfig 文件：

kubescape scan --kubeconfig cluster.conf

扫描特定的命名空间：

kubescape scan --include-namespaces development,staging,production

排除某些命名空间：

kubescape scan --exclude-namespaces kube-system,kube-public

部署前扫描本地 YAML/JSON 文件：

kubescape scan *.yaml

┌──[[e-mail]]-[~/ansible/helm]
└─$kubescape scan --enable-host-scan  kube-prometheus-stack.yaml  --format html  --output resout.html
[info] Kubescape scanner starting
[warning] in setContextMetadata. case: git-url; error: repository host 'gitee.com' not supported
[info] Downloading/Loading policy definitions
[success] Downloaded/Loaded policy
[info] Accessing local objects
[success] Done accessing local objects
[info] Scanning GitLocal
[success] Done scanning GitLocal^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^Controls: 55 (Failed: 31, Excluded: 0, Skipped: 0)
Failed Resources by Severity: Critical — 0, High — 34, Medium — 151, Low — 24+----------+--------------------------------------------------------------------------------------------+------------------+--------------------+---------------+--------------+
| SEVERITY |                                        CONTROL NAME                                        | FAILED RESOURCES | EXCLUDED RESOURCES | ALL RESOURCES | % RISK-SCORE |
+----------+--------------------------------------------------------------------------------------------+------------------+--------------------+---------------+--------------+
| High     | Resources memory limit and request                                                         |        7         |         0          |       7       |     100%     |
| High     | Resource limits                                                                            |        7         |         0          |       7       |     100%     |
| High     | List Kubernetes secrets                                                                    |        4         |         0          |       7       |     57%      |
。。。。。。。。。。。。。。。。。。。。。
| Medium   | Allow privilege escalation                                                                 |        6         |         0          |       7       |     86%      |
| Medium   | Ingress and Egress blocked                                                                 |        7         |         0          |       7       |     100%     |
。。。。。。。。。。。。。。。。。。。。。
| Medium   | CIS-5.4.1 Prefer using secrets as files over secrets as environment variables              |        1         |         0          |       7       |     14%      |
| Medium   | CIS-5.7.2 Ensure that the seccomp profile is set to docker/default in your pod definitions |        7         |         0          |       7       |     100%     |
| Medium   | CIS-5.7.4 The default namespace should not be used                                         |        56        |         0          |      61       |     92%      |
| Low      | Immutable container filesystem                                                             |        6         |         0          |       7       |     86%      |
| Low      | Configured readiness probe                                                                 |        5         |         0          |       7       |     71%      |
| Low      | Malicious admission controller (validating)                                                |        1         |         0          |       1       |     100%     |
| Low      | Pods in default namespace                                                                  |        7         |         0          |       7       |     100%     |
| Low      | Naked PODs                                                                                 |        1         |         0          |       1       |     100%     |
| Low      | Label usage for resources                                                                  |        3         |         0          |       7       |     43%      |
| Low      | K8s common labels usage                                                                    |        1         |         0          |       7       |     14%      |
+----------+--------------------------------------------------------------------------------------------+------------------+--------------------+---------------+--------------+
|          |                                      RESOURCE SUMMARY                                      |        66        |         0          |      72       |    43.26%    |
+----------+--------------------------------------------------------------------------------------------+------------------+--------------------+---------------+--------------+
FRAMEWORKS: ArmoBest (risk: 33.51), cis-v1.23-t1.0.1 (risk: 62.75), DevOpsBest (risk: 61.72), AllControls (risk: 36.46), MITRE (risk: 11.79), NSA (risk: 33.74)[success] Scan results saved. filename: resout.html~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Scan results have not been submitted: run kubescape with the '--account' flag
For more details: https://hub.armosec.io/docs/installing-kubescape?utm_campaign=Submit&utm_medium=CLI&utm_source=GitHub
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Run with '--verbose'/'-v' flag for detailed resources view┌──[[e-mail]]-[~/ansible/helm]
└─$l

从 Git 存储库扫描 Kubernetes 清单文件：

kubescape scan https://github.com/kubescape/kubescape

扫描 Helm 图表,kubescape 将加载默认的 VALUES 文件。

kubescape scan </path/to/directory>

扫描 Kustomize 目录,Kubescape 将使用 kustomize 文件生成 Kubernetes YAML 对象并扫描它们以确保安全。

kubescape scan </path/to/directory>

使用 NSA 框架扫描正在运行的 Kubernetes 集群：

kubescape scan framework nsa

使用 MITRE ATT&CK® 框架扫描正在运行的 Kubernetes 集群：

kubescape scan framework mitre

使用控件名称或控件 ID 扫描特定控件。请参阅控件列表。

kubescape scan control "Privileged container"

指定报告输出格式

JSON：

kubescape scan --format json --format-version v2 --output results.json

XML：

kubescape scan --format junit --output results.xml

PDF:

kubescape scan --format pdf --output results.pdf

普罗米修斯指标：

kubescape scan --format prometheus

HTML

kubescape scan --format html --output results.html

显示所有扫描到的资源（包括通过的资源）：

kubescape scan --verbose

Kubescape 在集群下安装

Kubescape 也可以在集群下安装，通过集群安装，可以在 cloud 里的 UI 界面查看具体扫描信息，集群中安装 Kubescape，先决条件

• 确保您拥有 Kubescape Cloud 帐户——如果没有，请在此处注册
• 您需要拥有对集群的安装权限（您应该能够创建 Deployments、CronJobs、ConfigMaps 和 Secrets）
• 你必须有 Kubectl 和 Helm

集群要求

• Kubescape 运算符组件至少需要 400Mib RAM 和 400m CPU

具体的安装可以参考下的教程： 需要注册 Kubescape Cloud，并且 只有的工作节点小于 10 个的时候才免费。

https://hub.armosec.io/docs/installation-of-armo-in-cluster#install-a-pre-registered-cluster

注册登录

这里登录完会弹出一个安装部署，安装安装部署安装即可

添加 Helm 源，并更新

┌──[[e-mail]]-[~/ansible/kubescape]
└─$helm repo add kubescape https://kubescape.github.io/helm-charts/
"kubescape" has been added to your repositories
┌──[[e-mail]]-[~/ansible/kubescape]
└─$helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "rancher-stable" chart repository
...Successfully got an update from the "botkube" chart repository
...Successfully got an update from the "awx-operator" chart repository
...Successfully got an update from the "kubescape" chart repository
Update Complete. ⎈Happy Helming!⎈

运行 charts

┌──[[e-mail]]-[~/ansible/kubescape]
└─$helm upgrade --install kubescape kubescape/kubescape-cloud-operator -n kubescape --create-namespace --set account=97f09924-0c06-42e4-bdad-5b333321af77 --set clusterName=`kubectl config current-context`
Release "kubescape" does not exist. Installing it now.
.........
......................
NAME: kubescape
LAST DEPLOYED: Sat Feb  4 10:03:36 2023
NAMESPACE: kubescape
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing kubescape-cloud-operator version 1.9.5.In a few minutes your scan results will be available in the following link:
https://cloud.armosec.io/config-scanning/kubernetes-admin-kubernetesYou can see and change the values of your's recurring configurations daily scan in the following link:
https://cloud.armosec.io/settings/assets/clusters/scheduled-scans?cluster=kubernetes-admin-kubernetes
> kubectl -n kubescape get cj kubescape-scheduler -o=jsonpath='{.metadata.name}{"\t"}{.spec.schedule}{"\n"}'You can see and change the values of your's recurring images daily scan in the following link:
https://cloud.armosec.io/settings/assets/images
> kubectl -n kubescape get cj kubevuln-scheduler -o=jsonpath='{.metadata.name}{"\t"}{.spec.schedule}{"\n"}'See you!!!
┌──[[e-mail]]-[~/ansible/kubescape]
└─$

运行完成，安装提示操作，点击刚才注册生成的页面。

验证 Kubescape 在集群中运行状态

┌──[[e-mail]]-[~/ansible/kubescape]
└─$kubectl -n kubescape get deployments.apps
NAME        READY   UP-TO-DATE   AVAILABLE   AGE
gateway     1/1     1            1           19m
kubescape   1/1     1            1           19m
kubevuln    1/1     1            1           19m
operator    1/1     1            1           19m

kubescape 会定期扫描

┌──[[e-mail]]-[~/ansible/kubescape]
└─$kubectl get cronjobs.batch -n kubescape
NAME                  SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
kubescape-scheduler   21 3 * * *    False     0        <none>          20m
kubevuln-scheduler    13 22 * * *   False     0        <none>          20m
┌──[[e-mail]]-[~/ansible/kubescape]
└─$

在登录的主页中可以看到集群和第一次扫描结果

安全合规性扫描信息

可以通过不同的维度来看

合规性的维度查看

集群 Pod/Deploy 的维度查看

具体的 合规配置信息查看

漏洞扫描

关于 kubescape 和小伙伴分享到这里，时间关系，没有深入太多。偶尔听大佬谈到，所以研究一下，有需要的小伙伴快去尝试吧。

博文部分内容参考

文中涉及参考链接内容版权归原作者所有，如有侵权请告知

---

https://github.com/kubescape/kubescape

https://www.redhat.com/en/blog/state-kubernetes-security-2022-1

https://betterprogramming.pub/image-vulnerability-scanning-for-optimal-kubernetes-security-c3ba933190ef

https://hub.armosec.io/docs/installation-of-armo-in-cluster#install-a-pre-registered-cluster

---

© 2018-2023 [e-mail]，All rights reserved. 保持署名-非商用-自由转载-相同方式共享(创意共享 3.0 许可证)