Kubelet方法具有在长时间高CPU使用率

我有编织CNI插件由3个节点的kubernetes群集：Kubelet方法具有在长时间高CPU使用率

1主节点（虚拟机）
2工人baremetall节点（4个核Xeon处理器超线程--8个逻辑节点）

问题是，top显示kubelet在第一个worker上有60-100％的CPU使用率。在journalctl -u kubelet我看到很多信息（数百每分钟）

May 19 09:57:38 kube-worker1 bash[3843]: E0519 09:57:38.075243 3843 docker_sandbox.go:205] Failed to stop sandbox "011cf10cf46dbc6bf2e11d1cb562af478eee21eba0c40521bf7af51ee5399640": Error response from daemon: {"message":"No such container: 011cf10cf46dbc6bf2e11d1cb562af478eee21eba0c40521bf7af51ee5399640"} 
May 19 09:57:38 kube-worker1 bash[3843]: E0519 09:57:38.075360 3843 remote_runtime.go:109] StopPodSandbox "011cf10cf46dbc6bf2e11d1cb562af478eee21eba0c40521bf7af51ee5399640" from runtime service failed: rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod "cron-task-2533948c46c1-p6kwb_namespace" network: CNI failed to retrieve network namespace path: Error: No such container: 011cf10cf46dbc6bf2e11d1cb562af478eee21eba0c40521bf7af51ee5399640 
May 19 09:57:38 kube-worker1 bash[3843]: E0519 09:57:38.075380 3843 kuberuntime_gc.go:138] Failed to stop sandbox "011cf10cf46dbc6bf2e11d1cb562af478eee21eba0c40521bf7af51ee5399640" before removing: rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod "cron-task-2533948c46c1-p6kwb_namespace" network: CNI failed to retrieve network namespace path: Error: No such container: 011cf10cf46dbc6bf2e11d1cb562af478eee21eba0c40521bf7af51ee5399640 
May 19 09:57:38 kube-worker1 bash[3843]: E0519 09:57:38.076549 3843 docker_sandbox.go:205] Failed to stop sandbox "0125de37634ef7f3aa852c999cfb5849750167b1e3d63293a085ceca416e4ebf": Error response from daemon: {"message":"No such container: 0125de37634ef7f3aa852c999cfb5849750167b1e3d63293a085ceca416e4ebf"} 
May 19 09:57:38 kube-worker1 bash[3843]: E0519 09:57:38.076654 3843 remote_runtime.go:109] StopPodSandbox "0125de37634ef7f3aa852c999cfb5849750167b1e3d63293a085ceca416e4ebf" from runtime service failed: rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod "cron-task-2533948c46c1-6g8jq_namespace" network: CNI failed to retrieve network namespace path: Error: No such container: 0125de37634ef7f3aa852c999cfb5849750167b1e3d63293a085ceca416e4ebf 
May 19 09:57:38 kube-worker1 bash[3843]: E0519 09:57:38.076676 3843 kuberuntime_gc.go:138] Failed to stop sandbox "0125de37634ef7f3aa852c999cfb5849750167b1e3d63293a085ceca416e4ebf" before removing: rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod "cron-task-2533948c46c1-6g8jq_namespace" network: CNI failed to retrieve network namespace path: Error: No such container: 0125de37634ef7f3aa852c999cfb5849750167b1e3d63293a085ceca416e4ebf 
May 19 09:57:38 kube-worker1 bash[3843]: E0519 09:57:38.079585 3843 docker_sandbox.go:205] Failed to stop sandbox "014135ede46ee45c176528da02782a38ded36bd10566f864c147ccb66a617772": Error response from daemon: {"message":"No such container: 014135ede46ee45c176528da02782a38ded36bd10566f864c147ccb66a617772"} 
May 19 09:57:38 kube-worker1 bash[3843]: E0519 09:57:38.079805 3843 remote_runtime.go:109] StopPodSandbox "014135ede46ee45c176528da02782a38ded36bd10566f864c147ccb66a617772" from runtime service failed: rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod "cron-task-2533948c46c1-r30cw_namespace" network: CNI failed to retrieve network namespace path: Error: No such container: 014135ede46ee45c176528da02782a38ded36bd10566f864c147ccb66a617772

它发生之后，在创建过程中失败的错误cronetes任务。我用--force删除了所有的豆荚，但kubelet仍然尝试删除它们。此外，我重新启动该工人kubelet没有结果。我如何与kubelet谈话忘记他们？

版本信息

Kubernetes v1.6.1 
Docker version 1.12.0, build 8eab29e 
Linux kube-worker1 4.4.0-72-generiC#93-Ubuntu SMP

集装箱舱单（不包括元数据）

job: 
    apiVersion: batch/v1 
    kind: Job 
    spec: 
     template: 
     spec: 
      containers: 
      - name: cron-task 
      image: docker.company.ru/image:v2.3.2 
      command: ["rake", "db:refresh_views"] 
      env: 
      - name: RAILS_ENV 
       value: namespace 
      - name: CONFIG_PATH 
       value: /config 
      volumeMounts: 
      - name: config 
       mountPath: /config 
      volumes: 
      - name: config 
      configMap: 
       name: task-conf 
      restartPolicy: Never

也没发现群集的ETCD名（2533948c46c1）的这一吊舱的一部分的任何提及。

来源

2017-05-19 user1802474

最后我找到了解决方案。
Kubelet存储有关的所有信息豆荚，在

/var/lib/dockershim/sandbox

在其上运行所以，当我ls该文件夹中，我发现文件中所有失踪吊舱。然后我删除这些文件和日志消息消失，并且CPU使用率恢复到正常值（即使没有kubelet重新启动）

来源

2017-05-25 06:46:01 user1802474

这似乎与Kubernetes 1.6.x中的Pods with hostNetwork=true cannot be removed (and generate errors) when using CNI问题有关。这些消息无论如何不是关键，但当你试图找到实际问题时，它当然很烦人。尝试使用最新版本的Kubernetes来缓解问题。

来源

2017-05-19 10:50:01 pagid

谢谢你的回答！但它看起来是一个不同的问题。我没有指定网络类型，所以我假设hostNetwork = false。我对吗？整个日志包含3种消息：1）StopPodSandbox，2）停止沙箱失败，3）MountVolume.SetUp成功。我应该提供任何其他信息吗？ – user1802474

我遇到了同样的问题，并且确实为此进行了分析并找到原因是kubelet pleg机制并删除'/ var/lib/dockershim/sandbox'做了魔术。

来源

2017-06-17 14:04:27 WizardCXY

Kubelet方法具有在长时间高CPU使用率

回答

相关问题