kubelet启动异常

问题原因

今天在mac上使用vagrant+vbox测试部署kubernetes时发现kubelet无法启动

查看kubelet日志发现报错信息如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
# journalctl -xeu kubelet
Mar 23 16:57:06 node1 kubelet[9386]: I0323 16:57:06.588641 9386 clientconn.go:796] ClientConn switching balancer to "pick_first"
Mar 23 16:57:06 node1 kubelet[9386]: I0323 16:57:06.588660 9386 balancer_conn_wrappers.go:131] pickfirstBalancer: HandleSubConnStateChange: 0xc00022c610, CONNECTING
Mar 23 16:57:06 node1 kubelet[9386]: I0323 16:57:06.588715 9386 balancer_conn_wrappers.go:131] pickfirstBalancer: HandleSubConnStateChange: 0xc00022c610, READY
Mar 23 16:57:07 node1 kubelet[9386]: E0323 16:57:07.568258 9386 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Node: Get https://mast
Mar 23 16:57:07 node1 kubelet[9386]: E0323 16:57:07.569987 9386 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/kubelet.go:442: Failed to list *v1.Service: Get https://m
Mar 23 16:57:07 node1 kubelet[9386]: E0323 16:57:07.573550 9386 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https
Mar 23 16:57:08 node1 kubelet[9386]: E0323 16:57:08.571026 9386 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Node: Get https://mast
Mar 23 16:57:08 node1 kubelet[9386]: E0323 16:57:08.571112 9386 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/kubelet.go:442: Failed to list *v1.Service: Get https://m
Mar 23 16:57:08 node1 kubelet[9386]: E0323 16:57:08.574628 9386 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https
Mar 23 16:57:09 node1 kubelet[9386]: E0323 16:57:09.573294 9386 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Node: Get https://mast
Mar 23 16:57:09 node1 kubelet[9386]: E0323 16:57:09.573367 9386 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/kubelet.go:442: Failed to list *v1.Service: Get https://m
Mar 23 16:57:09 node1 kubelet[9386]: E0323 16:57:09.575011 9386 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https
Mar 23 16:57:10 node1 kubelet[9386]: E0323 16:57:10.574665 9386 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Node: Get https://mast
Mar 23 16:57:10 node1 kubelet[9386]: E0323 16:57:10.575469 9386 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/kubelet.go:442: Failed to list *v1.Service: Get https://m
Mar 23 16:57:10 node1 kubelet[9386]: E0323 16:57:10.576079 9386 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https
Mar 23 16:57:11 node1 kubelet[9386]: W0323 16:57:11.574311 9386 cni.go:213] Unable to update cni config: No networks found in /etc/cni/net.d
Mar 23 16:57:11 node1 kubelet[9386]: E0323 16:57:11.575824 9386 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Node: Get https://mast
Mar 23 16:57:11 node1 kubelet[9386]: E0323 16:57:11.576473 9386 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/kubelet.go:442: Failed to list *v1.Service: Get https://m
Mar 23 16:57:11 node1 kubelet[9386]: E0323 16:57:11.577610 9386 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https
Mar 23 16:57:11 node1 kubelet[9386]: E0323 16:57:11.644966 9386 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain.
Mar 23 16:57:11 node1 kubelet[9386]: For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Mar 23 16:57:11 node1 kubelet[9386]: I0323 16:57:11.646745 9386 kuberuntime_manager.go:210] Container runtime docker initialized, version: 18.06.3-ce, apiVersion: 1.38.0
Mar 23 16:57:11 node1 kubelet[9386]: I0323 16:57:11.648488 9386 server.go:1055] Started kubelet
Mar 23 16:57:11 node1 kubelet[9386]: E0323 16:57:11.648894 9386 kubelet.go:1282] Image garbage collection failed once. Stats initialization may not have completed yet: f
Mar 23 16:57:11 node1 kubelet[9386]: I0323 16:57:11.649257 9386 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer
Mar 23 16:57:11 node1 kubelet[9386]: I0323 16:57:11.649275 9386 status_manager.go:152] Starting to sync pod status with apiserver
Mar 23 16:57:11 node1 kubelet[9386]: I0323 16:57:11.649287 9386 kubelet.go:1808] Starting kubelet main sync loop.
Mar 23 16:57:11 node1 kubelet[9386]: I0323 16:57:11.649296 9386 kubelet.go:1825] skipping pod synchronization - [container runtime status check may not have completed ye
Mar 23 16:57:11 node1 kubelet[9386]: I0323 16:57:11.649365 9386 server.go:141] Starting to listen on 0.0.0.0:10250
Mar 23 16:57:11 node1 kubelet[9386]: I0323 16:57:11.649786 9386 server.go:343] Adding debug handlers to kubelet server.
Mar 23 16:57:11 node1 kubelet[9386]: I0323 16:57:11.651043 9386 volume_manager.go:248] Starting Kubelet Volume Manager
Mar 23 16:57:11 node1 kubelet[9386]: I0323 16:57:11.655811 9386 desired_state_of_world_populator.go:130] Desired state populator starts to run
Mar 23 16:57:11 node1 kubelet[9386]: E0323 16:57:11.662116 9386 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.RuntimeClass: Get ht
Mar 23 16:57:11 node1 kubelet[9386]: E0323 16:57:11.662171 9386 controller.go:125] failed to ensure node lease exists, will retry in 200ms, error: Get https://master:644
Mar 23 16:57:11 node1 kubelet[9386]: E0323 16:57:11.662221 9386 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.CSIDriver: Get https
Mar 23 16:57:11 node1 kubelet[9386]: E0323 16:57:11.662260 9386 event.go:200] Unable to write event: 'Post https://master:6443/api/v1/namespaces/default/events: dial tcp
Mar 23 16:57:11 node1 kubelet[9386]: E0323 16:57:11.666432 9386 kubelet.go:2172] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady mes
Mar 23 16:57:11 node1 kubelet[9386]: I0323 16:57:11.719680 9386 kubelet_node_status.go:283] Setting node annotation to enable volume controller attach/detach
Mar 23 16:57:11 node1 kubelet[9386]: I0323 16:57:11.729810 9386 cpu_manager.go:155] [cpumanager] starting with none policy
Mar 23 16:57:11 node1 kubelet[9386]: I0323 16:57:11.729821 9386 cpu_manager.go:156] [cpumanager] reconciling every 10s
Mar 23 16:57:11 node1 kubelet[9386]: I0323 16:57:11.729827 9386 policy_none.go:42] [cpumanager] none policy: Start
Mar 23 16:57:11 node1 kubelet[9386]: E0323 16:57:11.748722 9386 node_container_manager_linux.go:50] Failed to create ["kubepods"] cgroup
Mar 23 16:57:11 node1 kubelet[9386]: F0323 16:57:11.748740 9386 kubelet.go:1359] Failed to start ContainerManager Cannot set property TasksAccounting, or unknown propert
Mar 23 16:57:11 node1 systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
Mar 23 16:57:11 node1 systemd[1]: Unit kubelet.service entered failed state.
Mar 23 16:57:11 node1 systemd[1]: kubelet.service failed.

解决方法

在kubernetes的issues中找到对应信息并且有解决方案https://github.com/kubernetes/kubernetes/issues/76820

升级systemd即可解决

1
yum -y upgrade systemd

当前网速较慢或者你使用的浏览器不支持博客特定功能,请尝试刷新或换用Chrome、Firefox等现代浏览器