kubeadmin kube-apiserver Exited 始终起不来查因记录
kubeadmin kube-apiserver Exited 始终起不来查因记录
[root@k8s-master01 log]# crictl ps -a
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
b7af23a98302e fce326961ae2d 16 seconds ago Running etcd 29 16fc6f83a01d2 etcd-k8s-master01
1d0efa5c0c12d a31e1d84401e6 About a minute ago Exited kube-apiserver 1693 dfc4c0a0c3e03 kube-apiserver-k8s-master01
275f08ddab851 fce326961ae2d 6 minutes ago Exited etcd 28 16fc6f83a01d2 etcd-k8s-master01
b11025cbc4661 5d7c5dfd3ba18 6 hours ago Running kube-controller-manager 27 0ff05b544ff48 kube-controller-manager-k8s-master01
4db4688c2687f dafd8ad70b156 30 hours ago Running kube-scheduler 25 5f4d13cedf450 kube-scheduler-k8s-master01
b311bf0e66852 54637cb36d4a1 7 days ago Running calico-node 0 ff2f4ac3783bb calico-node-2zqhn
108695e1af006 a1a5060fe43dc 9 days ago Running kuboard 2 7bee3baf06a62 kuboard-cc79974cd-t9jth
536a8cdfb0a9b 115053965e86b 9 days ago Running metrics-scraper 2 046881f3feea3 metrics-scraper-7f4896c5d7-6w6ld
c91c3382c9c9d 556768f31eb1d 9 days ago Running kube-proxy 6 ce658d774a03b kube-proxy-gsv75
[root@k8s-master01 log]#
查日志
cat /var/log/messages|grep kube-apiserver|grep -i errorFeb 16 12:08:16 k8s-master01 kubelet: E0216 12:08:16.192310 8996 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-apiserver\" with CrashLoopBackOff: \"back-off 40s restarting failed container=kube-apiserver pod=kube-apiserver-k8s-master01_kube-system(d6b90b54ef1ec678fb3557edd6baf627)\"" pod="kube-system/kube-apiserver-k8s-master01" podUID=d6b90b54ef1ec678fb3557edd6baf627
由于 kube-apiserver 起不来
kubectl describe
kubectl logs
之类查问题的命令都用不了
用的containerd.service 容器时
crictl logs 命令可用
[root@k8s-master01 log]# crictl ps -a
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
b3c010e0082be a31e1d84401e6 39 seconds ago Exited kube-apiserver 1701 dfc4c0a0c3e03 kube-apiserver-k8s-master01
401eba6ca6507 fce326961ae2d 2 minutes ago Running etcd 36 16fc6f83a01d2 etcd-k8s-master01
b11025cbc4661 5d7c5dfd3ba18 7 hours ago Running kube-controller-manager 27 0ff05b544ff48 kube-controller-manager-k8s-master01
4db4688c2687f dafd8ad70b156 31 hours ago Running kube-scheduler 25 5f4d13cedf450 kube-scheduler-k8s-master01
b311bf0e66852 54637cb36d4a1 7 days ago Running calico-node 0 ff2f4ac3783bb calico-node-2zqhn
108695e1af006 a1a5060fe43dc 9 days ago Running kuboard 2 7bee3baf06a62 kuboard-cc79974cd-t9jth
536a8cdfb0a9b 115053965e86b 9 days ago Running metrics-scraper 2 046881f3feea3 metrics-scraper-7f4896c5d7-6w6ld
c91c3382c9c9d 556768f31eb1d 9 days ago Running kube-proxy 6 ce658d774a03b kube-proxy-gsv75
[root@k8s-master01 log]# crictl logs b3c010e0082be
I0216 04:41:58.479545 1 server.go:555] external host was not specified, using 192.168.40.240
I0216 04:41:58.480162 1 server.go:163] Version: v1.26.0
I0216 04:41:58.480188 1 server.go:165] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0216 04:41:58.662208 1 shared_informer.go:273] Waiting for caches to sync for node_authorizer
I0216 04:41:58.662957 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0216 04:41:58.662977 1 plugins.go:161] Loaded 12 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionPolicy,ValidatingAdmissionWebhook,ResourceQuota.
E0216 04:42:18.665556 1 run.go:74] "command failed" err="context deadline exceeded"
W0216 04:42:18.665571 1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {"Addr": "127.0.0.1:2379","ServerName": "127.0.0.1","Attributes": null,"BalancerAttributes": null,"Type": 0,"Metadata": null
}. Err: connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
[root@k8s-master01 log]#
貌似连接2379 etcd服务端口异常
crictl logs 860b2d24e75c3 #etcd容器的id
{"level":"info","ts":"2023-02-16T04:48:39.238Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e7a1aee40b9b8621 [logterm: 107, index: 5657524] sent MsgPreVote request to 9331fa9e272a9c3f at term 107"}
{"level":"warn","ts":"2023-02-16T04:48:40.151Z","caller":"etcdserver/server.go:2075","msg":"failed to publish local member to cluster through raft","local-member-id":"e7a1aee40b9b8621","local-member-attributes":"{Name:k8s-master01 ClientURLs:[https://192.168.40.240:2379]}","request-path":"/0/members/e7a1aee40b9b8621/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"info","ts":"2023-02-16T04:48:40.438Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e7a1aee40b9b8621 is starting a new election at term 107"}
{"level":"info","ts":"2023-02-16T04:48:40.438Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e7a1aee40b9b8621 became pre-candidate at term 107"}
{"level":"info","ts":"2023-02-16T04:48:40.438Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e7a1aee40b9b8621 received MsgPreVoteResp from e7a1aee40b9b8621 at term 107"}
{"level":"info","ts":"2023-02-16T04:48:40.438Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e7a1aee40b9b8621 [logterm: 107, index: 5657524] sent MsgPreVote request to 5db16f545fa302b7 at term 107"}
{"level":"info","ts":"2023-02-16T04:48:40.438Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e7a1aee40b9b8621 [logterm: 107, index: 5657524] sent MsgPreVote request to 9331fa9e272a9c3f at term 107"}
{"level":"warn","ts":"2023-02-16T04:48:40.447Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"}
{"level":"warn","ts":"2023-02-16T04:48:40.448Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503}
.......
{"level":"warn","ts":"2023-02-16T06:55:15.139Z","caller":"etcdserver/server.go:2075","msg":"failed to publish local member to cluster through raft","local-member-id":"e7a1aee40b9b8621","local-member-attributes":"{Name:k8s-master01 ClientURLs:[https://192.168.40.240:2379]}","request-path":"/0/members/e7a1aee40b9b8621/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
[root@k8s-master01 log]#
从日志看，像是etcd内部异常
参考
http://www.caotama.com/1864029.html
受启发，etcd备节点需要先启动,主节点接入备节点才能完整工作。
查备节点，发现因为多网卡IP是动态
[root@k8s-master03 manifests]# crictl ps -a
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
a62d091cb77d6 a31e1d84401e6 24 seconds ago Exited kube-apiserver 92 f32dfab0c6d61 kube-apiserver-k8s-master03
580675496bfae fce326961ae2d 21 minutes ago Exited etcd 88 40fa395fc5f61 etcd-k8s-master03
d68f4e7158815 5d7c5dfd3ba18 7 hours ago Running kube-controller-manager 29 e46f277014232 kube-controller-manager-k8s-master03
7e42849ff064e dafd8ad70b156 7 hours ago Running kube-scheduler 28 95d3e74185619 kube-scheduler-k8s-master03
69a554f8f249e 5d7c5dfd3ba18 3 days ago Exited kube-controller-manager 28 d435a4f2a2550 kube-controller-manager-k8s-master03
927c7330986d4 dafd8ad70b156 3 days ago Exited kube-scheduler 27 6bd230494824d kube-scheduler-k8s-master03
2b61030c4f724 5185b96f0becf 13 days ago Exited coredns 1 417cff0be932c coredns-567c556887-gqxc2
e62bfc4d92bf2 54637cb36d4a1 13 days ago Exited calico-node 2 65cb679894ec0 calico-node-4qgnf
e419b9d4ed335 54637cb36d4a1 13 days ago Exited mount-bpffs 0 65cb679894ec0 calico-node-4qgnf
bce9d5d9b2faa 628dd70880410 13 days ago Exited install-cni 0 65cb679894ec0 calico-node-4qgnf
1508b3b2344e8 556768f31eb1d 13 days ago Exited kube-proxy 2 4d7e52b0ee4b5 kube-proxy-qn6lx
ee1622dfb7d36 628dd70880410 13 days ago Exited upgrade-ipam 2 65cb679894ec0 calico-node-4qgnf
[root@k8s-master03 manifests]#
[root@k8s-master03 manifests]#
[root@k8s-master03 manifests]# crictl logs 580675496bfae
{"level":"info","ts":"2023-02-16T08:48:38.698Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd","--advertise-client-urls=https://172.19.217.2:2379","--cert-file=/etc/kubernetes/pki/etcd/server.crt","--client-cert-auth=true","--data-dir=/var/lib/eted","--experimental-initial-corrupt-check=true","--experimental-watch-progress-notify-interval=5s","--initial-advertise-peer-urls=https://172.19.217.2:2380","--initial-cluster=k8s-master03=https://172.19.217.2:2380,k8s-master02=https://172.19.217.32:2380,k8s-master01=https://192.168.40.240:2380","--initial-cluster-state=existing","--key-file=/etc/kubernetes/pki/etcd/server.key","--listen-client-urls=https://127.0.0.1:2379,https://172.19.217.2:2379","--listen-metrics-urls=http://127.0.0.1:2381","--listen-peer-urls=https://172.19.217.2:2380","--name=k8s-master03","--peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt","--peer-client-cert-auth=true","--peer-key-file=/etc/kubernetes/pki/etcd/peer.key","--peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt","--snapshot-count=10000","--trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt"]}
{"level":"info","ts":"2023-02-16T08:48:38.698Z","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/var/lib/eted","dir-type":"member"}
{"level":"info","ts":"2023-02-16T08:48:38.698Z","caller":"embed/etcd.go:124","msg":"configuring peer listeners","listen-peer-urls":["https://172.19.217.2:2380"]}
{"level":"info","ts":"2023-02-16T08:48:38.698Z","caller":"embed/etcd.go:484","msg":"starting with peer TLS","tls-info":"cert = /etc/kubernetes/pki/etcd/peer.crt, key = /etc/kubernetes/pki/etcd/peer.key, client-cert=, client-key=, trusted-ca = /etc/kubernetes/pki/etcd/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":[]}
{"level":"info","ts":"2023-02-16T08:48:38.698Z","caller":"embed/etcd.go:373","msg":"closing etcd server","name":"k8s-master03","data-dir":"/var/lib/eted","advertise-peer-urls":["https://172.19.217.2:2380"],"advertise-client-urls":["https://172.19.217.2:2379"]}
{"level":"info","ts":"2023-02-16T08:48:38.698Z","caller":"embed/etcd.go:375","msg":"closed etcd server","name":"k8s-master03","data-dir":"/var/lib/eted","advertise-peer-urls":["https://172.19.217.2:2380"],"advertise-client-urls":["https://172.19.217.2:2379"]}
{"level":"fatal","ts":"2023-02-16T08:48:38.698Z","caller":"etcdmain/etcd.go:204","msg":"discovery failed","error":"listen tcp 172.19.217.2:2380: bind: cannot assign requested address","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:204\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:32\nruntime.main\n\truntime/proc.go:225"}
[root@k8s-master03 manifests]#
https://172.19.217.2:2380
应该是因为多网卡动态获取的ip，eth1网卡为 ssh连接专用网卡，但eth0-IP已经变化
将eth0网卡静态成该地址172.19.217.2。
修复 。
相关文章：
kubeadmin kube-apiserver Exited 始终起不来查因记录
kubeadmin kube-apiserver Exited 始终起不来查因记录 [rootk8s-master01 log]# crictl ps -a CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD b7af23a98302e …...
论文投稿指南——中文核心期刊推荐（工程材料学）
【前言】 🚀 想发论文怎么办？手把手教你论文如何投稿！那么，首先要搞懂投稿目标——论文期刊 🎄 在期刊论文的分布中，存在一种普遍现象：即对于某一特定的学科或专业来说，少数期刊所含…...
【动态规划】背包问题题型及方法归纳
背包问题的种类 背包问题是在规定背包容量为j的前提下，每个物品对应的体积为v[i]，价值为w[i]，从物品0到物品i中选择物品放入背包中，找出符合某种要求的价值。 （1）背包问题种类 01背包：每种物…...
全球十大资质正规外汇期货平台排行榜（最新版汇总）
外汇期货简称为FxFut，是“Forex Futures”的缩写，是在集中形式的期货交易所内，交易双方通过公开叫价，以某种非本国货币买进或卖出另一种非本国货币，并签订一个在未来的某一日期根据协议价格交割标准数量外汇的合约。 …...
使用Paramiko时遇到的一些问题
目录 1.背景 2.问题合集 1）“bash: command not found” 2）Paramiko中正常的输入，却到了stderr，而stdout是空 3）命令实际是alias 1.背景 在自动化脚本中，使用了库Paramiko，远程SSH到后台服…...
数据预处理（无量纲化、缺失值、分类特征、连续特征）
文章目录1. 无量纲化1.1 sklearn.preprocessing.MinMaxScaler1.2 sklearn.preprocessing.StandardScaler2. 缺失值3. 分类型特征4. 连续型特征数据挖掘的五大流程包括：获取数据数据预处理特征工程建模上线 其中，数据预处理中常用的方法包括数据标准化和归…...
【C#基础】C# 运算符总结
序号系列文章2【C#基础】C# 基础语法解析3【C#基础】C# 数据类型总结4【C#基础】C# 变量和常量的使用文章目录前言运算符1，算术运算符2，布尔逻辑运算符3，位运算符4，关系运算符5，赋值运算符6，其他运算符7&am…...
存储性能软件加速库（SPDK）
存储性能软件加速库SPDK存储加速存储性能软件加速库（SPDK）SPDK NVMe驱动1.用户态驱动1）UIO2）VFIOIOMMU（I/O Memory Management Unit）3）用户态DMA4）大页（Hugepage…...
微服务（五）—— 服务注册中心Consul
一、引入依赖 <dependency><groupId>org.springframework.cloud</groupId><artifactId>spring-cloud-starter-consul-discovery</artifactId></dependency>二、配置yml文件 server:port: 8006spring:application:name: cloud-payment-con…...
冷冻电镜 - ChimeraX Density Map 密度图 操作
欢迎关注我的CSDN：https://spike.blog.csdn.net/ 本文地址：https://blog.csdn.net/caroline_wendy/article/details/129055160 由冷冻电镜所生成的Volume，需要观察其内部结构，使用ChimeraX进行操作。 加载Volumes，例如my_volume.mrc 效果如下： 高斯滤波 在命令行(Co…...
Matlab 点云旋转之轴角式
文章目录 一、简介二、实现代码三、实现效果参考资料一、简介 三维空间中表示旋转的方法有很多种，轴角式是其中非常经典的一种表示方式。虽然欧拉角表示旋转的方法很是常用，但欧拉角存在着万向锁这个问题，因此轴角式旋转在旋转使用中更为合适。其原理也很是明了，如下所述：…...
2023美赛数学建模资料思路模型
美赛我们为大家准备了大量的资料，我们会在比赛期间给大家分析美题目和相关的思路 全文都是干货，大家仔细阅读，资料文末自取！ 首先我们来看美赛23年题型的一个变化： 美赛23年题目变化： A题：连…...
Nginx配置HTTP强制跳转到HTTPS
https 访问我们的测试域名 https://www.xxx.com 站点，但是当我们直接在浏览器地址栏中直接输入 www.xxx.com 的时候却发现进入的是 http 协议的网站，这与我们的初衷不一致。由于浏览器默认访问域名使用的是80端口，而当我们使用SSL证书后&…...
从实现到原理，聊聊Java中的SPI动态扩展
原创：微信公众号 码农参上，欢迎分享，转载请保留出处。 八股文背多了，相信大家都听说过一个词，SPI扩展。 有的面试官就很喜欢问这个问题，SpringBoot的自动装配是如何实现的？ 基本上，…...
3、MySQL字符集
1.MySQL字符集和校验规则 字符集：是一套符号和编码的规则校验规则：是对该套符号和编码的校验，定义字符的排序和比较规则,其中是否区分大小写，跟校验规则有关。2.查看字符集方法 netstat -lntup |grep 3306 tcp6 0 0 :::3306 :::* …...
大漠插件最新中文易语言模块7.2302
模块名称:大漠插件中文模块最新通用7.2302模块简介:大漠插件中文模块最新通用7.2302模块特色:原翻译:花老板完善命令备注:易生易世本人花费一个月时间才将命令完善了插件的备注说明.且用且珍惜去掉了大漠插件定制版类.因为没用.模块特色:什么是中文模块?大漠插件模块是由大漠类…...
极客大挑战 2021
题量很大，收获挺多，持续时间也长，据说结束之后会再持续一段时间，然后题目会开源。 WEB Dark 暗网签到，难以置信 Welcome2021 改个请求方法会提示你文件，再进去就好了 babysql 直接把请求包扔sqlmap里&…...
C#开发的OpenRA加载文件的管理
C#开发的OpenRA加载文件的管理 在前面我们分析了mod.yaml文件，发现里面有很多文件列表， 比如下像下面的文件： Packages: ~^SupportDir|Content/cnc ~^SupportDir|Content/cnc/movies ^EngineDir $cnc: cnc ^EngineDir|mods/common: common ~speech.mix ~conquer.mix ~sounds…...
SSM实现文件上传
目录 SSM实现文件上传 1、修改from表单请求方式改为post，添加属性 2、修改springmvc配置文件，添加一下配置 3、后端方法 SSM实现文件上传 1、修改from表单请求方式改为post，添加属性： enctype"multipart/form-data"…...
OPENCV计算机视觉开发实践-图像的基本概念
1.图像与图形: 图像->客观世界的反映,图与像之结合 图->物体透射光与反射光的分布 像->人的视觉得对图的认识 图像->通过照相,摄像,扫描产生. 图形->通过数学规则产生,或者具有一定规则的图案.用一组符号或线条表示性质. 2.数字图像: 数字图像->称数码图像或…...
【Oracle APEX开发小技巧12】
有如下需求： 有一个问题反馈页面，要实现在apex页面展示能直观看到反馈时间超过7天未处理的数据，方便管理员及时处理反馈。 我的方法：直接将逻辑写在SQL中，这样可以直接在页面展示 完整代码： SELECTSF.FE…...
Python：操作 Excel 折叠
💖亲爱的技术爱好者们，热烈欢迎来到 Kant2048 的博客！我是 Thomas Kant，很开心能在CSDN上与你们相遇～💖 本博客的精华专栏： 【自动化测试】 【测试经验】 【人工智能】 【Python】 Python 操作 Excel 系列 读取单元格数据按行写入设置行高和列宽自动调整行高和列宽水平…...
ffmpeg（四）：滤镜命令
FFmpeg 的滤镜命令是用于音视频处理中的强大工具，可以完成剪裁、缩放、加水印、调色、合成、旋转、模糊、叠加字幕等复杂的操作。其核心语法格式一般如下： ffmpeg -i input.mp4 -vf "滤镜参数" output.mp4或者带音频滤镜： ffmpeg…...
大数据学习（132）-HIve数据分析
🍋🍋大数据学习🍋🍋 🔥系列专栏： 👑哲学语录: 用力所能及，改变世界。 💖如果觉得博主的文章还不错的话，请点赞👍收藏⭐️留言Ǵ…...
使用 SymPy 进行向量和矩阵的高级操作
在科学计算和工程领域，向量和矩阵操作是解决问题的核心技能之一。Python 的 SymPy 库提供了强大的符号计算功能，能够高效地处理向量和矩阵的各种操作。本文将深入探讨如何使用 SymPy 进行向量和矩阵的创建、合并以及维度拓展等操作，并通过具体…...
10-Oracle 23 ai Vector Search 概述和参数
一、Oracle AI Vector Search 概述 企业和个人都在尝试各种AI，使用客户端或是内部自己搭建集成大模型的终端，加速与大型语言模型（LLM）的结合，同时使用检索增强生成（Retrieval Augmented Generation &#…...
HarmonyOS运动开发：如何用mpchart绘制运动配速图表
##鸿蒙核心技术##运动开发##Sensor Service Kit（传感器服务）# 前言 在运动类应用中，运动数据的可视化是提升用户体验的重要环节。通过直观的图表展示运动过程中的关键数据，如配速、距离、卡路里消耗等，用户可以更清晰…...
Mysql中select查询语句的执行过程
目录 1、介绍 1.1、组件介绍 1.2、Sql执行顺序 2、执行流程 2.1. 连接与认证 2.2. 查询缓存 2.3. 语法解析（Parser） 2.4、执行sql 1. 预处理（Preprocessor） 2. 查询优化器（Optimizer） 3. 执行器…...
CVPR2025重磅突破：AnomalyAny框架实现单样本生成逼真异常数据，破解视觉检测瓶颈！
本文介绍了一种名为AnomalyAny的创新框架，该方法利用Stable Diffusion的强大生成能力，仅需单个正常样本和文本描述，即可生成逼真且多样化的异常样本，有效解决了视觉异常检测中异常样本稀缺的难题，为工业质检、医疗影像…...
MySQL 主从同步异常处理
阅读原文：https://www.xiaozaoshu.top/articles/mysql-m-s-update-pk MySQL 做双主，遇到的这个错误： Could not execute Update_rows event on table ... Error_code: 1032是 MySQL 主从复制时的经典错误之一，通常表示ÿ…...