kubeadmin kube-apiserver Exited 始终起不来查因记录
kubeadmin kube-apiserver Exited 始终起不来查因记录
[root@k8s-master01 log]# crictl ps -a
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
b7af23a98302e fce326961ae2d 16 seconds ago Running etcd 29 16fc6f83a01d2 etcd-k8s-master01
1d0efa5c0c12d a31e1d84401e6 About a minute ago Exited kube-apiserver 1693 dfc4c0a0c3e03 kube-apiserver-k8s-master01
275f08ddab851 fce326961ae2d 6 minutes ago Exited etcd 28 16fc6f83a01d2 etcd-k8s-master01
b11025cbc4661 5d7c5dfd3ba18 6 hours ago Running kube-controller-manager 27 0ff05b544ff48 kube-controller-manager-k8s-master01
4db4688c2687f dafd8ad70b156 30 hours ago Running kube-scheduler 25 5f4d13cedf450 kube-scheduler-k8s-master01
b311bf0e66852 54637cb36d4a1 7 days ago Running calico-node 0 ff2f4ac3783bb calico-node-2zqhn
108695e1af006 a1a5060fe43dc 9 days ago Running kuboard 2 7bee3baf06a62 kuboard-cc79974cd-t9jth
536a8cdfb0a9b 115053965e86b 9 days ago Running metrics-scraper 2 046881f3feea3 metrics-scraper-7f4896c5d7-6w6ld
c91c3382c9c9d 556768f31eb1d 9 days ago Running kube-proxy 6 ce658d774a03b kube-proxy-gsv75
[root@k8s-master01 log]#
查日志
cat /var/log/messages|grep kube-apiserver|grep -i errorFeb 16 12:08:16 k8s-master01 kubelet: E0216 12:08:16.192310 8996 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-apiserver\" with CrashLoopBackOff: \"back-off 40s restarting failed container=kube-apiserver pod=kube-apiserver-k8s-master01_kube-system(d6b90b54ef1ec678fb3557edd6baf627)\"" pod="kube-system/kube-apiserver-k8s-master01" podUID=d6b90b54ef1ec678fb3557edd6baf627
由于 kube-apiserver 起不来
kubectl describe
kubectl logs
之类查问题的命令都用不了
用的containerd.service 容器时
crictl logs 命令可用
[root@k8s-master01 log]# crictl ps -a
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
b3c010e0082be a31e1d84401e6 39 seconds ago Exited kube-apiserver 1701 dfc4c0a0c3e03 kube-apiserver-k8s-master01
401eba6ca6507 fce326961ae2d 2 minutes ago Running etcd 36 16fc6f83a01d2 etcd-k8s-master01
b11025cbc4661 5d7c5dfd3ba18 7 hours ago Running kube-controller-manager 27 0ff05b544ff48 kube-controller-manager-k8s-master01
4db4688c2687f dafd8ad70b156 31 hours ago Running kube-scheduler 25 5f4d13cedf450 kube-scheduler-k8s-master01
b311bf0e66852 54637cb36d4a1 7 days ago Running calico-node 0 ff2f4ac3783bb calico-node-2zqhn
108695e1af006 a1a5060fe43dc 9 days ago Running kuboard 2 7bee3baf06a62 kuboard-cc79974cd-t9jth
536a8cdfb0a9b 115053965e86b 9 days ago Running metrics-scraper 2 046881f3feea3 metrics-scraper-7f4896c5d7-6w6ld
c91c3382c9c9d 556768f31eb1d 9 days ago Running kube-proxy 6 ce658d774a03b kube-proxy-gsv75
[root@k8s-master01 log]# crictl logs b3c010e0082be
I0216 04:41:58.479545 1 server.go:555] external host was not specified, using 192.168.40.240
I0216 04:41:58.480162 1 server.go:163] Version: v1.26.0
I0216 04:41:58.480188 1 server.go:165] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0216 04:41:58.662208 1 shared_informer.go:273] Waiting for caches to sync for node_authorizer
I0216 04:41:58.662957 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0216 04:41:58.662977 1 plugins.go:161] Loaded 12 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionPolicy,ValidatingAdmissionWebhook,ResourceQuota.
E0216 04:42:18.665556 1 run.go:74] "command failed" err="context deadline exceeded"
W0216 04:42:18.665571 1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {"Addr": "127.0.0.1:2379","ServerName": "127.0.0.1","Attributes": null,"BalancerAttributes": null,"Type": 0,"Metadata": null
}. Err: connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
[root@k8s-master01 log]#
貌似连接2379 etcd服务端口异常
crictl logs 860b2d24e75c3 #etcd容器的id
{"level":"info","ts":"2023-02-16T04:48:39.238Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e7a1aee40b9b8621 [logterm: 107, index: 5657524] sent MsgPreVote request to 9331fa9e272a9c3f at term 107"}
{"level":"warn","ts":"2023-02-16T04:48:40.151Z","caller":"etcdserver/server.go:2075","msg":"failed to publish local member to cluster through raft","local-member-id":"e7a1aee40b9b8621","local-member-attributes":"{Name:k8s-master01 ClientURLs:[https://192.168.40.240:2379]}","request-path":"/0/members/e7a1aee40b9b8621/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"info","ts":"2023-02-16T04:48:40.438Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e7a1aee40b9b8621 is starting a new election at term 107"}
{"level":"info","ts":"2023-02-16T04:48:40.438Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e7a1aee40b9b8621 became pre-candidate at term 107"}
{"level":"info","ts":"2023-02-16T04:48:40.438Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e7a1aee40b9b8621 received MsgPreVoteResp from e7a1aee40b9b8621 at term 107"}
{"level":"info","ts":"2023-02-16T04:48:40.438Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e7a1aee40b9b8621 [logterm: 107, index: 5657524] sent MsgPreVote request to 5db16f545fa302b7 at term 107"}
{"level":"info","ts":"2023-02-16T04:48:40.438Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e7a1aee40b9b8621 [logterm: 107, index: 5657524] sent MsgPreVote request to 9331fa9e272a9c3f at term 107"}
{"level":"warn","ts":"2023-02-16T04:48:40.447Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"}
{"level":"warn","ts":"2023-02-16T04:48:40.448Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503}
.......
{"level":"warn","ts":"2023-02-16T06:55:15.139Z","caller":"etcdserver/server.go:2075","msg":"failed to publish local member to cluster through raft","local-member-id":"e7a1aee40b9b8621","local-member-attributes":"{Name:k8s-master01 ClientURLs:[https://192.168.40.240:2379]}","request-path":"/0/members/e7a1aee40b9b8621/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
[root@k8s-master01 log]#
从日志看，像是etcd内部异常
参考
http://www.caotama.com/1864029.html
受启发，etcd备节点需要先启动,主节点接入备节点才能完整工作。
查备节点，发现因为多网卡IP是动态
[root@k8s-master03 manifests]# crictl ps -a
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
a62d091cb77d6 a31e1d84401e6 24 seconds ago Exited kube-apiserver 92 f32dfab0c6d61 kube-apiserver-k8s-master03
580675496bfae fce326961ae2d 21 minutes ago Exited etcd 88 40fa395fc5f61 etcd-k8s-master03
d68f4e7158815 5d7c5dfd3ba18 7 hours ago Running kube-controller-manager 29 e46f277014232 kube-controller-manager-k8s-master03
7e42849ff064e dafd8ad70b156 7 hours ago Running kube-scheduler 28 95d3e74185619 kube-scheduler-k8s-master03
69a554f8f249e 5d7c5dfd3ba18 3 days ago Exited kube-controller-manager 28 d435a4f2a2550 kube-controller-manager-k8s-master03
927c7330986d4 dafd8ad70b156 3 days ago Exited kube-scheduler 27 6bd230494824d kube-scheduler-k8s-master03
2b61030c4f724 5185b96f0becf 13 days ago Exited coredns 1 417cff0be932c coredns-567c556887-gqxc2
e62bfc4d92bf2 54637cb36d4a1 13 days ago Exited calico-node 2 65cb679894ec0 calico-node-4qgnf
e419b9d4ed335 54637cb36d4a1 13 days ago Exited mount-bpffs 0 65cb679894ec0 calico-node-4qgnf
bce9d5d9b2faa 628dd70880410 13 days ago Exited install-cni 0 65cb679894ec0 calico-node-4qgnf
1508b3b2344e8 556768f31eb1d 13 days ago Exited kube-proxy 2 4d7e52b0ee4b5 kube-proxy-qn6lx
ee1622dfb7d36 628dd70880410 13 days ago Exited upgrade-ipam 2 65cb679894ec0 calico-node-4qgnf
[root@k8s-master03 manifests]#
[root@k8s-master03 manifests]#
[root@k8s-master03 manifests]# crictl logs 580675496bfae
{"level":"info","ts":"2023-02-16T08:48:38.698Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd","--advertise-client-urls=https://172.19.217.2:2379","--cert-file=/etc/kubernetes/pki/etcd/server.crt","--client-cert-auth=true","--data-dir=/var/lib/eted","--experimental-initial-corrupt-check=true","--experimental-watch-progress-notify-interval=5s","--initial-advertise-peer-urls=https://172.19.217.2:2380","--initial-cluster=k8s-master03=https://172.19.217.2:2380,k8s-master02=https://172.19.217.32:2380,k8s-master01=https://192.168.40.240:2380","--initial-cluster-state=existing","--key-file=/etc/kubernetes/pki/etcd/server.key","--listen-client-urls=https://127.0.0.1:2379,https://172.19.217.2:2379","--listen-metrics-urls=http://127.0.0.1:2381","--listen-peer-urls=https://172.19.217.2:2380","--name=k8s-master03","--peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt","--peer-client-cert-auth=true","--peer-key-file=/etc/kubernetes/pki/etcd/peer.key","--peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt","--snapshot-count=10000","--trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt"]}
{"level":"info","ts":"2023-02-16T08:48:38.698Z","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/var/lib/eted","dir-type":"member"}
{"level":"info","ts":"2023-02-16T08:48:38.698Z","caller":"embed/etcd.go:124","msg":"configuring peer listeners","listen-peer-urls":["https://172.19.217.2:2380"]}
{"level":"info","ts":"2023-02-16T08:48:38.698Z","caller":"embed/etcd.go:484","msg":"starting with peer TLS","tls-info":"cert = /etc/kubernetes/pki/etcd/peer.crt, key = /etc/kubernetes/pki/etcd/peer.key, client-cert=, client-key=, trusted-ca = /etc/kubernetes/pki/etcd/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":[]}
{"level":"info","ts":"2023-02-16T08:48:38.698Z","caller":"embed/etcd.go:373","msg":"closing etcd server","name":"k8s-master03","data-dir":"/var/lib/eted","advertise-peer-urls":["https://172.19.217.2:2380"],"advertise-client-urls":["https://172.19.217.2:2379"]}
{"level":"info","ts":"2023-02-16T08:48:38.698Z","caller":"embed/etcd.go:375","msg":"closed etcd server","name":"k8s-master03","data-dir":"/var/lib/eted","advertise-peer-urls":["https://172.19.217.2:2380"],"advertise-client-urls":["https://172.19.217.2:2379"]}
{"level":"fatal","ts":"2023-02-16T08:48:38.698Z","caller":"etcdmain/etcd.go:204","msg":"discovery failed","error":"listen tcp 172.19.217.2:2380: bind: cannot assign requested address","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:204\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:32\nruntime.main\n\truntime/proc.go:225"}
[root@k8s-master03 manifests]#
https://172.19.217.2:2380
应该是因为多网卡动态获取的ip，eth1网卡为 ssh连接专用网卡，但eth0-IP已经变化
将eth0网卡静态成该地址172.19.217.2。
修复 。
相关文章：
kubeadmin kube-apiserver Exited 始终起不来查因记录
kubeadmin kube-apiserver Exited 始终起不来查因记录 [rootk8s-master01 log]# crictl ps -a CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD b7af23a98302e …...
论文投稿指南——中文核心期刊推荐（工程材料学）
【前言】 🚀 想发论文怎么办？手把手教你论文如何投稿！那么，首先要搞懂投稿目标——论文期刊 🎄 在期刊论文的分布中，存在一种普遍现象：即对于某一特定的学科或专业来说，少数期刊所含…...
【动态规划】背包问题题型及方法归纳
背包问题的种类 背包问题是在规定背包容量为j的前提下，每个物品对应的体积为v[i]，价值为w[i]，从物品0到物品i中选择物品放入背包中，找出符合某种要求的价值。 （1）背包问题种类 01背包：每种物…...
全球十大资质正规外汇期货平台排行榜（最新版汇总）
外汇期货简称为FxFut，是“Forex Futures”的缩写，是在集中形式的期货交易所内，交易双方通过公开叫价，以某种非本国货币买进或卖出另一种非本国货币，并签订一个在未来的某一日期根据协议价格交割标准数量外汇的合约。 …...
使用Paramiko时遇到的一些问题
目录 1.背景 2.问题合集 1）“bash: command not found” 2）Paramiko中正常的输入，却到了stderr，而stdout是空 3）命令实际是alias 1.背景 在自动化脚本中，使用了库Paramiko，远程SSH到后台服…...
数据预处理（无量纲化、缺失值、分类特征、连续特征）
文章目录1. 无量纲化1.1 sklearn.preprocessing.MinMaxScaler1.2 sklearn.preprocessing.StandardScaler2. 缺失值3. 分类型特征4. 连续型特征数据挖掘的五大流程包括：获取数据数据预处理特征工程建模上线 其中，数据预处理中常用的方法包括数据标准化和归…...
【C#基础】C# 运算符总结
序号系列文章2【C#基础】C# 基础语法解析3【C#基础】C# 数据类型总结4【C#基础】C# 变量和常量的使用文章目录前言运算符1，算术运算符2，布尔逻辑运算符3，位运算符4，关系运算符5，赋值运算符6，其他运算符7&am…...
存储性能软件加速库（SPDK）
存储性能软件加速库SPDK存储加速存储性能软件加速库（SPDK）SPDK NVMe驱动1.用户态驱动1）UIO2）VFIOIOMMU（I/O Memory Management Unit）3）用户态DMA4）大页（Hugepage…...
微服务（五）—— 服务注册中心Consul
一、引入依赖 <dependency><groupId>org.springframework.cloud</groupId><artifactId>spring-cloud-starter-consul-discovery</artifactId></dependency>二、配置yml文件 server:port: 8006spring:application:name: cloud-payment-con…...
冷冻电镜 - ChimeraX Density Map 密度图 操作
欢迎关注我的CSDN：https://spike.blog.csdn.net/ 本文地址：https://blog.csdn.net/caroline_wendy/article/details/129055160 由冷冻电镜所生成的Volume，需要观察其内部结构，使用ChimeraX进行操作。 加载Volumes，例如my_volume.mrc 效果如下： 高斯滤波 在命令行(Co…...
Matlab 点云旋转之轴角式
文章目录 一、简介二、实现代码三、实现效果参考资料一、简介 三维空间中表示旋转的方法有很多种，轴角式是其中非常经典的一种表示方式。虽然欧拉角表示旋转的方法很是常用，但欧拉角存在着万向锁这个问题，因此轴角式旋转在旋转使用中更为合适。其原理也很是明了，如下所述：…...
2023美赛数学建模资料思路模型
美赛我们为大家准备了大量的资料，我们会在比赛期间给大家分析美题目和相关的思路 全文都是干货，大家仔细阅读，资料文末自取！ 首先我们来看美赛23年题型的一个变化： 美赛23年题目变化： A题：连…...
Nginx配置HTTP强制跳转到HTTPS
https 访问我们的测试域名 https://www.xxx.com 站点，但是当我们直接在浏览器地址栏中直接输入 www.xxx.com 的时候却发现进入的是 http 协议的网站，这与我们的初衷不一致。由于浏览器默认访问域名使用的是80端口，而当我们使用SSL证书后&…...
从实现到原理，聊聊Java中的SPI动态扩展
原创：微信公众号 码农参上，欢迎分享，转载请保留出处。 八股文背多了，相信大家都听说过一个词，SPI扩展。 有的面试官就很喜欢问这个问题，SpringBoot的自动装配是如何实现的？ 基本上，…...
3、MySQL字符集
1.MySQL字符集和校验规则 字符集：是一套符号和编码的规则校验规则：是对该套符号和编码的校验，定义字符的排序和比较规则,其中是否区分大小写，跟校验规则有关。2.查看字符集方法 netstat -lntup |grep 3306 tcp6 0 0 :::3306 :::* …...
大漠插件最新中文易语言模块7.2302
模块名称:大漠插件中文模块最新通用7.2302模块简介:大漠插件中文模块最新通用7.2302模块特色:原翻译:花老板完善命令备注:易生易世本人花费一个月时间才将命令完善了插件的备注说明.且用且珍惜去掉了大漠插件定制版类.因为没用.模块特色:什么是中文模块?大漠插件模块是由大漠类…...
极客大挑战 2021
题量很大，收获挺多，持续时间也长，据说结束之后会再持续一段时间，然后题目会开源。 WEB Dark 暗网签到，难以置信 Welcome2021 改个请求方法会提示你文件，再进去就好了 babysql 直接把请求包扔sqlmap里&…...
C#开发的OpenRA加载文件的管理
C#开发的OpenRA加载文件的管理 在前面我们分析了mod.yaml文件，发现里面有很多文件列表， 比如下像下面的文件： Packages: ~^SupportDir|Content/cnc ~^SupportDir|Content/cnc/movies ^EngineDir $cnc: cnc ^EngineDir|mods/common: common ~speech.mix ~conquer.mix ~sounds…...
SSM实现文件上传
目录 SSM实现文件上传 1、修改from表单请求方式改为post，添加属性 2、修改springmvc配置文件，添加一下配置 3、后端方法 SSM实现文件上传 1、修改from表单请求方式改为post，添加属性： enctype"multipart/form-data"…...
OPENCV计算机视觉开发实践-图像的基本概念
1.图像与图形: 图像->客观世界的反映,图与像之结合 图->物体透射光与反射光的分布 像->人的视觉得对图的认识 图像->通过照相,摄像,扫描产生. 图形->通过数学规则产生,或者具有一定规则的图案.用一组符号或线条表示性质. 2.数字图像: 数字图像->称数码图像或…...
渲染学进阶内容——模型
最近在写模组的时候发现渲染器里面离不开模型的定义，在渲染的第二篇文章中简单的讲解了一下关于模型部分的内容，其实不管是方块还是方块实体，都离不开模型的内容 🧱 一、CubeListBuilder 功能解析 CubeListBuilder 是 Minecraft Java 版模型系统的核心构建器，用于动态创…...
Nuxt.js 中的路由配置详解
Nuxt.js 通过其内置的路由系统简化了应用的路由配置，使得开发者可以轻松地管理页面导航和 URL 结构。路由配置主要涉及页面组件的组织、动态路由的设置以及路由元信息的配置。 自动路由生成 Nuxt.js 会根据 pages 目录下的文件结构自动生成路由配置。每个文件都会对…...
深入解析C++中的extern关键字：跨文件共享变量与函数的终极指南
🚀 C extern 关键字深度解析：跨文件编程的终极指南 📅 更新时间：2025年6月5日 🏷️ 标签：C | extern关键字 | 多文件编程 | 链接与声明 | 现代C 文章目录 前言🔥一、extern 是什么？&…...
Linux --进程控制
本文从以下五个方面来初步认识进程控制： 目录 进程创建 进程终止 进程等待 进程替换 模拟实现一个微型shell 进程创建 在Linux系统中我们可以在一个进程使用系统调用fork()来创建子进程，创建出来的进程就是子进程，原来的进程为父进程。…...
详细解析)
Caliper 负载(Workload)详细解析
Caliper 负载(Workload)详细解析 负载(Workload)是 Caliper 性能测试的核心部分，它定义了测试期间要执行的具体合约调用行为和交易模式。下面我将全面深入地讲解负载的各个方面。 一、负载模块基本结构 一个典型的负载模块(如 workload.js)包含以下基本结构： use strict;/…...
mac：大模型系列测试
0 MAC 前几天经过学生优惠以及国补17K入手了mac studio,然后这两天亲自测试其模型行运用能力如何，是否支持微调、推理速度等能力。下面进入正文。 1 mac 与 unsloth 按照下面的进行安装以及测试，是可以跑通文章里面的代码。训练速度也是很快的。 注意…...
【FTP】ftp文件传输会丢包吗？批量几百个文件传输，有一些文件没有传输完整，如何解决？
FTP（File Transfer Protocol）本身是一个基于 TCP 的协议，理论上不会丢包。但 FTP 文件传输过程中仍可能出现文件不完整、丢失或损坏的情况，主要原因包括： ✅ 一、FTP传输可能“丢包”或文件不完整的原因 原因描述网络…...
【WebSocket】SpringBoot项目中使用WebSocket
1. 导入坐标 如果springboot父工程没有加入websocket的起步依赖，添加它的坐标的时候需要带上版本号。 <dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-websocket</artifactId> </dep…...
[USACO23FEB] Bakery S
题目描述 Bessie 开了一家面包店! 在她的面包店里，Bessie 有一个烤箱，可以在 t C t_C tC 的时间内生产一块饼干或在 t M t_M tM 单位时间内生产一块松糕。 ( 1 ≤ t C , t M ≤ 10 9 ) (1 \le t_C,t_M \le 10^9) (1≤tC,tM≤109)。由于空间…...
goreplay
1.github地址 https://github.com/buger/goreplay 2.简单介绍 GoReplay 是一个开源的网络监控工具，可以记录用户的实时流量并将其用于镜像、负载测试、监控和详细分析。 3.出现背景 随着应用程序的增长，测试它所需的工作量也会呈指数级增长。GoRepl…...