kubeadmin kube-apiserver Exited 始终起不来查因记录
kubeadmin kube-apiserver Exited 始终起不来查因记录
[root@k8s-master01 log]# crictl ps -a
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
b7af23a98302e fce326961ae2d 16 seconds ago Running etcd 29 16fc6f83a01d2 etcd-k8s-master01
1d0efa5c0c12d a31e1d84401e6 About a minute ago Exited kube-apiserver 1693 dfc4c0a0c3e03 kube-apiserver-k8s-master01
275f08ddab851 fce326961ae2d 6 minutes ago Exited etcd 28 16fc6f83a01d2 etcd-k8s-master01
b11025cbc4661 5d7c5dfd3ba18 6 hours ago Running kube-controller-manager 27 0ff05b544ff48 kube-controller-manager-k8s-master01
4db4688c2687f dafd8ad70b156 30 hours ago Running kube-scheduler 25 5f4d13cedf450 kube-scheduler-k8s-master01
b311bf0e66852 54637cb36d4a1 7 days ago Running calico-node 0 ff2f4ac3783bb calico-node-2zqhn
108695e1af006 a1a5060fe43dc 9 days ago Running kuboard 2 7bee3baf06a62 kuboard-cc79974cd-t9jth
536a8cdfb0a9b 115053965e86b 9 days ago Running metrics-scraper 2 046881f3feea3 metrics-scraper-7f4896c5d7-6w6ld
c91c3382c9c9d 556768f31eb1d 9 days ago Running kube-proxy 6 ce658d774a03b kube-proxy-gsv75
[root@k8s-master01 log]#
查日志
cat /var/log/messages|grep kube-apiserver|grep -i errorFeb 16 12:08:16 k8s-master01 kubelet: E0216 12:08:16.192310 8996 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-apiserver\" with CrashLoopBackOff: \"back-off 40s restarting failed container=kube-apiserver pod=kube-apiserver-k8s-master01_kube-system(d6b90b54ef1ec678fb3557edd6baf627)\"" pod="kube-system/kube-apiserver-k8s-master01" podUID=d6b90b54ef1ec678fb3557edd6baf627
由于 kube-apiserver 起不来
kubectl describe
kubectl logs
之类查问题的命令都用不了
用的containerd.service 容器时
crictl logs 命令可用
[root@k8s-master01 log]# crictl ps -a
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
b3c010e0082be a31e1d84401e6 39 seconds ago Exited kube-apiserver 1701 dfc4c0a0c3e03 kube-apiserver-k8s-master01
401eba6ca6507 fce326961ae2d 2 minutes ago Running etcd 36 16fc6f83a01d2 etcd-k8s-master01
b11025cbc4661 5d7c5dfd3ba18 7 hours ago Running kube-controller-manager 27 0ff05b544ff48 kube-controller-manager-k8s-master01
4db4688c2687f dafd8ad70b156 31 hours ago Running kube-scheduler 25 5f4d13cedf450 kube-scheduler-k8s-master01
b311bf0e66852 54637cb36d4a1 7 days ago Running calico-node 0 ff2f4ac3783bb calico-node-2zqhn
108695e1af006 a1a5060fe43dc 9 days ago Running kuboard 2 7bee3baf06a62 kuboard-cc79974cd-t9jth
536a8cdfb0a9b 115053965e86b 9 days ago Running metrics-scraper 2 046881f3feea3 metrics-scraper-7f4896c5d7-6w6ld
c91c3382c9c9d 556768f31eb1d 9 days ago Running kube-proxy 6 ce658d774a03b kube-proxy-gsv75
[root@k8s-master01 log]# crictl logs b3c010e0082be
I0216 04:41:58.479545 1 server.go:555] external host was not specified, using 192.168.40.240
I0216 04:41:58.480162 1 server.go:163] Version: v1.26.0
I0216 04:41:58.480188 1 server.go:165] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0216 04:41:58.662208 1 shared_informer.go:273] Waiting for caches to sync for node_authorizer
I0216 04:41:58.662957 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0216 04:41:58.662977 1 plugins.go:161] Loaded 12 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionPolicy,ValidatingAdmissionWebhook,ResourceQuota.
E0216 04:42:18.665556 1 run.go:74] "command failed" err="context deadline exceeded"
W0216 04:42:18.665571 1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {"Addr": "127.0.0.1:2379","ServerName": "127.0.0.1","Attributes": null,"BalancerAttributes": null,"Type": 0,"Metadata": null
}. Err: connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
[root@k8s-master01 log]#
貌似连接2379 etcd服务端口异常
crictl logs 860b2d24e75c3 #etcd容器的id
{"level":"info","ts":"2023-02-16T04:48:39.238Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e7a1aee40b9b8621 [logterm: 107, index: 5657524] sent MsgPreVote request to 9331fa9e272a9c3f at term 107"}
{"level":"warn","ts":"2023-02-16T04:48:40.151Z","caller":"etcdserver/server.go:2075","msg":"failed to publish local member to cluster through raft","local-member-id":"e7a1aee40b9b8621","local-member-attributes":"{Name:k8s-master01 ClientURLs:[https://192.168.40.240:2379]}","request-path":"/0/members/e7a1aee40b9b8621/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"info","ts":"2023-02-16T04:48:40.438Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e7a1aee40b9b8621 is starting a new election at term 107"}
{"level":"info","ts":"2023-02-16T04:48:40.438Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e7a1aee40b9b8621 became pre-candidate at term 107"}
{"level":"info","ts":"2023-02-16T04:48:40.438Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e7a1aee40b9b8621 received MsgPreVoteResp from e7a1aee40b9b8621 at term 107"}
{"level":"info","ts":"2023-02-16T04:48:40.438Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e7a1aee40b9b8621 [logterm: 107, index: 5657524] sent MsgPreVote request to 5db16f545fa302b7 at term 107"}
{"level":"info","ts":"2023-02-16T04:48:40.438Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e7a1aee40b9b8621 [logterm: 107, index: 5657524] sent MsgPreVote request to 9331fa9e272a9c3f at term 107"}
{"level":"warn","ts":"2023-02-16T04:48:40.447Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"}
{"level":"warn","ts":"2023-02-16T04:48:40.448Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503}
.......
{"level":"warn","ts":"2023-02-16T06:55:15.139Z","caller":"etcdserver/server.go:2075","msg":"failed to publish local member to cluster through raft","local-member-id":"e7a1aee40b9b8621","local-member-attributes":"{Name:k8s-master01 ClientURLs:[https://192.168.40.240:2379]}","request-path":"/0/members/e7a1aee40b9b8621/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
[root@k8s-master01 log]#
从日志看，像是etcd内部异常
参考
http://www.caotama.com/1864029.html
受启发，etcd备节点需要先启动,主节点接入备节点才能完整工作。
查备节点，发现因为多网卡IP是动态
[root@k8s-master03 manifests]# crictl ps -a
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
a62d091cb77d6 a31e1d84401e6 24 seconds ago Exited kube-apiserver 92 f32dfab0c6d61 kube-apiserver-k8s-master03
580675496bfae fce326961ae2d 21 minutes ago Exited etcd 88 40fa395fc5f61 etcd-k8s-master03
d68f4e7158815 5d7c5dfd3ba18 7 hours ago Running kube-controller-manager 29 e46f277014232 kube-controller-manager-k8s-master03
7e42849ff064e dafd8ad70b156 7 hours ago Running kube-scheduler 28 95d3e74185619 kube-scheduler-k8s-master03
69a554f8f249e 5d7c5dfd3ba18 3 days ago Exited kube-controller-manager 28 d435a4f2a2550 kube-controller-manager-k8s-master03
927c7330986d4 dafd8ad70b156 3 days ago Exited kube-scheduler 27 6bd230494824d kube-scheduler-k8s-master03
2b61030c4f724 5185b96f0becf 13 days ago Exited coredns 1 417cff0be932c coredns-567c556887-gqxc2
e62bfc4d92bf2 54637cb36d4a1 13 days ago Exited calico-node 2 65cb679894ec0 calico-node-4qgnf
e419b9d4ed335 54637cb36d4a1 13 days ago Exited mount-bpffs 0 65cb679894ec0 calico-node-4qgnf
bce9d5d9b2faa 628dd70880410 13 days ago Exited install-cni 0 65cb679894ec0 calico-node-4qgnf
1508b3b2344e8 556768f31eb1d 13 days ago Exited kube-proxy 2 4d7e52b0ee4b5 kube-proxy-qn6lx
ee1622dfb7d36 628dd70880410 13 days ago Exited upgrade-ipam 2 65cb679894ec0 calico-node-4qgnf
[root@k8s-master03 manifests]#
[root@k8s-master03 manifests]#
[root@k8s-master03 manifests]# crictl logs 580675496bfae
{"level":"info","ts":"2023-02-16T08:48:38.698Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd","--advertise-client-urls=https://172.19.217.2:2379","--cert-file=/etc/kubernetes/pki/etcd/server.crt","--client-cert-auth=true","--data-dir=/var/lib/eted","--experimental-initial-corrupt-check=true","--experimental-watch-progress-notify-interval=5s","--initial-advertise-peer-urls=https://172.19.217.2:2380","--initial-cluster=k8s-master03=https://172.19.217.2:2380,k8s-master02=https://172.19.217.32:2380,k8s-master01=https://192.168.40.240:2380","--initial-cluster-state=existing","--key-file=/etc/kubernetes/pki/etcd/server.key","--listen-client-urls=https://127.0.0.1:2379,https://172.19.217.2:2379","--listen-metrics-urls=http://127.0.0.1:2381","--listen-peer-urls=https://172.19.217.2:2380","--name=k8s-master03","--peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt","--peer-client-cert-auth=true","--peer-key-file=/etc/kubernetes/pki/etcd/peer.key","--peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt","--snapshot-count=10000","--trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt"]}
{"level":"info","ts":"2023-02-16T08:48:38.698Z","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/var/lib/eted","dir-type":"member"}
{"level":"info","ts":"2023-02-16T08:48:38.698Z","caller":"embed/etcd.go:124","msg":"configuring peer listeners","listen-peer-urls":["https://172.19.217.2:2380"]}
{"level":"info","ts":"2023-02-16T08:48:38.698Z","caller":"embed/etcd.go:484","msg":"starting with peer TLS","tls-info":"cert = /etc/kubernetes/pki/etcd/peer.crt, key = /etc/kubernetes/pki/etcd/peer.key, client-cert=, client-key=, trusted-ca = /etc/kubernetes/pki/etcd/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":[]}
{"level":"info","ts":"2023-02-16T08:48:38.698Z","caller":"embed/etcd.go:373","msg":"closing etcd server","name":"k8s-master03","data-dir":"/var/lib/eted","advertise-peer-urls":["https://172.19.217.2:2380"],"advertise-client-urls":["https://172.19.217.2:2379"]}
{"level":"info","ts":"2023-02-16T08:48:38.698Z","caller":"embed/etcd.go:375","msg":"closed etcd server","name":"k8s-master03","data-dir":"/var/lib/eted","advertise-peer-urls":["https://172.19.217.2:2380"],"advertise-client-urls":["https://172.19.217.2:2379"]}
{"level":"fatal","ts":"2023-02-16T08:48:38.698Z","caller":"etcdmain/etcd.go:204","msg":"discovery failed","error":"listen tcp 172.19.217.2:2380: bind: cannot assign requested address","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:204\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:32\nruntime.main\n\truntime/proc.go:225"}
[root@k8s-master03 manifests]#
https://172.19.217.2:2380
应该是因为多网卡动态获取的ip，eth1网卡为 ssh连接专用网卡，但eth0-IP已经变化
将eth0网卡静态成该地址172.19.217.2。
修复 。
相关文章：
kubeadmin kube-apiserver Exited 始终起不来查因记录
kubeadmin kube-apiserver Exited 始终起不来查因记录 [rootk8s-master01 log]# crictl ps -a CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD b7af23a98302e …...
论文投稿指南——中文核心期刊推荐（工程材料学）
【前言】 🚀 想发论文怎么办？手把手教你论文如何投稿！那么，首先要搞懂投稿目标——论文期刊 🎄 在期刊论文的分布中，存在一种普遍现象：即对于某一特定的学科或专业来说，少数期刊所含…...
【动态规划】背包问题题型及方法归纳
背包问题的种类 背包问题是在规定背包容量为j的前提下，每个物品对应的体积为v[i]，价值为w[i]，从物品0到物品i中选择物品放入背包中，找出符合某种要求的价值。 （1）背包问题种类 01背包：每种物…...
全球十大资质正规外汇期货平台排行榜（最新版汇总）
外汇期货简称为FxFut，是“Forex Futures”的缩写，是在集中形式的期货交易所内，交易双方通过公开叫价，以某种非本国货币买进或卖出另一种非本国货币，并签订一个在未来的某一日期根据协议价格交割标准数量外汇的合约。 …...
使用Paramiko时遇到的一些问题
目录 1.背景 2.问题合集 1）“bash: command not found” 2）Paramiko中正常的输入，却到了stderr，而stdout是空 3）命令实际是alias 1.背景 在自动化脚本中，使用了库Paramiko，远程SSH到后台服…...
数据预处理（无量纲化、缺失值、分类特征、连续特征）
文章目录1. 无量纲化1.1 sklearn.preprocessing.MinMaxScaler1.2 sklearn.preprocessing.StandardScaler2. 缺失值3. 分类型特征4. 连续型特征数据挖掘的五大流程包括：获取数据数据预处理特征工程建模上线 其中，数据预处理中常用的方法包括数据标准化和归…...
【C#基础】C# 运算符总结
序号系列文章2【C#基础】C# 基础语法解析3【C#基础】C# 数据类型总结4【C#基础】C# 变量和常量的使用文章目录前言运算符1，算术运算符2，布尔逻辑运算符3，位运算符4，关系运算符5，赋值运算符6，其他运算符7&am…...
存储性能软件加速库（SPDK）
存储性能软件加速库SPDK存储加速存储性能软件加速库（SPDK）SPDK NVMe驱动1.用户态驱动1）UIO2）VFIOIOMMU（I/O Memory Management Unit）3）用户态DMA4）大页（Hugepage…...
微服务（五）—— 服务注册中心Consul
一、引入依赖 <dependency><groupId>org.springframework.cloud</groupId><artifactId>spring-cloud-starter-consul-discovery</artifactId></dependency>二、配置yml文件 server:port: 8006spring:application:name: cloud-payment-con…...
冷冻电镜 - ChimeraX Density Map 密度图 操作
欢迎关注我的CSDN：https://spike.blog.csdn.net/ 本文地址：https://blog.csdn.net/caroline_wendy/article/details/129055160 由冷冻电镜所生成的Volume，需要观察其内部结构，使用ChimeraX进行操作。 加载Volumes，例如my_volume.mrc 效果如下： 高斯滤波 在命令行(Co…...
Matlab 点云旋转之轴角式
文章目录 一、简介二、实现代码三、实现效果参考资料一、简介 三维空间中表示旋转的方法有很多种，轴角式是其中非常经典的一种表示方式。虽然欧拉角表示旋转的方法很是常用，但欧拉角存在着万向锁这个问题，因此轴角式旋转在旋转使用中更为合适。其原理也很是明了，如下所述：…...
2023美赛数学建模资料思路模型
美赛我们为大家准备了大量的资料，我们会在比赛期间给大家分析美题目和相关的思路 全文都是干货，大家仔细阅读，资料文末自取！ 首先我们来看美赛23年题型的一个变化： 美赛23年题目变化： A题：连…...
Nginx配置HTTP强制跳转到HTTPS
https 访问我们的测试域名 https://www.xxx.com 站点，但是当我们直接在浏览器地址栏中直接输入 www.xxx.com 的时候却发现进入的是 http 协议的网站，这与我们的初衷不一致。由于浏览器默认访问域名使用的是80端口，而当我们使用SSL证书后&…...
从实现到原理，聊聊Java中的SPI动态扩展
原创：微信公众号 码农参上，欢迎分享，转载请保留出处。 八股文背多了，相信大家都听说过一个词，SPI扩展。 有的面试官就很喜欢问这个问题，SpringBoot的自动装配是如何实现的？ 基本上，…...
3、MySQL字符集
1.MySQL字符集和校验规则 字符集：是一套符号和编码的规则校验规则：是对该套符号和编码的校验，定义字符的排序和比较规则,其中是否区分大小写，跟校验规则有关。2.查看字符集方法 netstat -lntup |grep 3306 tcp6 0 0 :::3306 :::* …...
大漠插件最新中文易语言模块7.2302
模块名称:大漠插件中文模块最新通用7.2302模块简介:大漠插件中文模块最新通用7.2302模块特色:原翻译:花老板完善命令备注:易生易世本人花费一个月时间才将命令完善了插件的备注说明.且用且珍惜去掉了大漠插件定制版类.因为没用.模块特色:什么是中文模块?大漠插件模块是由大漠类…...
极客大挑战 2021
题量很大，收获挺多，持续时间也长，据说结束之后会再持续一段时间，然后题目会开源。 WEB Dark 暗网签到，难以置信 Welcome2021 改个请求方法会提示你文件，再进去就好了 babysql 直接把请求包扔sqlmap里&…...
C#开发的OpenRA加载文件的管理
C#开发的OpenRA加载文件的管理 在前面我们分析了mod.yaml文件，发现里面有很多文件列表， 比如下像下面的文件： Packages: ~^SupportDir|Content/cnc ~^SupportDir|Content/cnc/movies ^EngineDir $cnc: cnc ^EngineDir|mods/common: common ~speech.mix ~conquer.mix ~sounds…...
SSM实现文件上传
目录 SSM实现文件上传 1、修改from表单请求方式改为post，添加属性 2、修改springmvc配置文件，添加一下配置 3、后端方法 SSM实现文件上传 1、修改from表单请求方式改为post，添加属性： enctype"multipart/form-data"…...
OPENCV计算机视觉开发实践-图像的基本概念
1.图像与图形: 图像->客观世界的反映,图与像之结合 图->物体透射光与反射光的分布 像->人的视觉得对图的认识 图像->通过照相,摄像,扫描产生. 图形->通过数学规则产生,或者具有一定规则的图案.用一组符号或线条表示性质. 2.数字图像: 数字图像->称数码图像或…...
linux 错误码总结
1，错误码的概念与作用 在Linux系统中，错误码是系统调用或库函数在执行失败时返回的特定数值，用于指示具体的错误类型。这些错误码通过全局变量errno来存储和传递，errno由操作系统维护，保存最近一次发生的错误信息。值得注意的是，errno的值在每次系统调用或函数调用失败时…...
【Java_EE】Spring MVC
目录 Spring Web MVC 编辑注解 RestController RequestMapping RequestParam RequestParam RequestBody PathVariable RequestPart 参数传递 注意事项 编辑参数重命名 RequestParam 编辑编辑传递集合 RequestParam 传递JSON数据 编辑RequestBody …...
JUC笔记(上)-复习 涉及死锁 volatile synchronized CAS 原子操作
一、上下文切换 即使单核CPU也可以进行多线程执行代码，CPU会给每个线程分配CPU时间片来实现这个机制。时间片非常短，所以CPU会不断地切换线程执行，从而让我们感觉多个线程是同时执行的。时间片一般是十几毫秒(ms)。通过时间片分配算法执行。…...
智能分布式爬虫的数据处理流水线优化：基于深度强化学习的数据质量控制
在数字化浪潮席卷全球的今天，数据已成为企业和研究机构的核心资产。智能分布式爬虫作为高效的数据采集工具，在大规模数据获取中发挥着关键作用。然而，传统的数据处理流水线在面对复杂多变的网络环境和海量异构数据时，常出现数据质…...
Springboot社区养老保险系统小程序
一、前言 随着我国经济迅速发展，人们对手机的需求越来越大，各种手机软件也都在被广泛应用，但是对于手机进行数据信息管理，对于手机的各种软件也是备受用户的喜爱，社区养老保险系统小程序被用户普遍使用，为方…...
html css js网页制作成品——HTML+CSS榴莲商城网页设计（4页）附源码
目录 一、👨🎓网站题目 二、✍️网站描述 三、📚网站介绍 四、🌐网站效果 五、🪓 代码实现 🧱HTML 六、🥇 如何让学习不再盲目 七、🎁更多干货 一、👨…...
Yolov8 目标检测蒸馏学习记录
yolov8系列模型蒸馏基本流程，代码下载：这里本人提交了一个demo:djdll/Yolov8_Distillation: Yolov8轻量化_蒸馏代码实现 在轻量化模型设计中，**知识蒸馏（Knowledge Distillation）**被广泛应用，作为提升模型…...
C/C++ 中附加包含目录、附加库目录与附加依赖项详解
在 C/C 编程的编译和链接过程中，附加包含目录、附加库目录和附加依赖项是三个至关重要的设置，它们相互配合，确保程序能够正确引用外部资源并顺利构建。虽然在学习过程中，这些概念容易让人混淆，但深入理解它们的作用和联…...
MySQL 8.0 事务全面讲解
以下是一个结合两次回答的 MySQL 8.0 事务全面讲解，涵盖了事务的核心概念、操作示例、失败回滚、隔离级别、事务性 DDL 和 XA 事务等内容，并修正了查看隔离级别的命令。 MySQL 8.0 事务全面讲解 一、事务的核心概念（ACID） 事务是…...
苹果AI眼镜：从“工具”到“社交姿态”的范式革命——重新定义AI交互入口的未来机会
在2025年的AI硬件浪潮中，苹果AI眼镜（Apple Glasses）正在引发一场关于“人机交互形态”的深度思考。它并非简单地替代AirPods或Apple Watch，而是开辟了一个全新的、日常可接受的AI入口。其核心价值不在于功能的堆叠，而在于如何通过形态设计打破社交壁垒，成为用户“全天佩戴…...