filebeat初始化失败,出现10.96.0.1:443 i/o超时错误

i7uaboj4  于 2021-06-15  发布在  ElasticSearch
关注(0)|答案(1)|浏览(907)

在我的k8s集群中,节点重启后filebeat连接失败。其他k8s节点工作正常。
来自filebeat pod的日志:

  1. 2020-08-30T03:18:58.770Z ERROR kubernetes/util.go:90 kubernetes: Querying for pod failed with error: performing request: Get https://10.96.0.1:443/api/v1/namespaces/monitoring/pods/filebeat-gfg5l: dial tcp 10.96.0.1:443: i/o timeout
  2. 2020-08-30T03:18:58.770Z INFO kubernetes/watcher.go:180 kubernetes: Performing a resource sync for *v1.PodList
  3. 2020-08-30T03:19:28.771Z ERROR kubernetes/watcher.go:183 kubernetes: Performing a resource sync err performing request: Get https://10.96.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout for *v1.PodList
  4. 2020-08-30T03:19:28.771Z INFO instance/beat.go:357 filebeat stopped.
  5. 2020-08-30T03:19:28.771Z ERROR instance/beat.go:800 Exiting: error initializing publisher: error initializing processors: performing request: Get https://10.96.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
  6. Exiting: error initializing publisher: error initializing processors: performing request: Get https://10.96.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

出现错误并重复pod重新启动。我也重新启动了这个节点,但它没有工作。
filebeat版本为6.5.2,使用守护程序集进行部署。有没有类似的已知问题?
除filebeat外,在该节点上工作的所有pod都没有问题。
更新:

  1. apiVersion: v1
  2. data:
  3. filebeat.yml: |-
  4. filebeat.inputs:
  5. - type: docker
  6. multiline.pattern: '^[[:space:]]+'
  7. multiline.negate: false
  8. multiline.match: after
  9. symlinks: true
  10. cri.parse_flags: true
  11. containers:
  12. ids: [""]
  13. path: "/var/log/containers"
  14. processors:
  15. - decode_json_fields:
  16. fields: ["message"]
  17. process_array: false
  18. max_depth: 1
  19. target: message_json
  20. overwrite_keys: false
  21. when:
  22. contains:
  23. source: "/var/log/containers/app"
  24. - add_kubernetes_metadata:
  25. in_cluster: true
  26. default_matchers.enabled: false
  27. matchers:
  28. - logs_path:
  29. logs_path: /var/log/containers/
  30. output:
  31. logstash:
  32. hosts:
  33. - logstash:5044
  34. kind: ConfigMap
  35. metadata:
  36. creationTimestamp: "2020-01-06T09:31:31Z"
  37. labels:
  38. k8s-app: filebeat
  39. name: filebeat-config
  40. namespace: monitoring
  41. resourceVersion: "6797684985"
  42. selfLink: /api/v1/namespaces/monitoring/configmaps/filebeat-config
  43. uid: 52d86bbb-3067-11ea-89c6-246e96da5c9c
fumotvh3

fumotvh31#

这个 add_kubernetes_metadata 查询失败 https://10.96.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost&resourceVersion=0 . 正如上面讨论的结果,这可以通过重新启动解决临时网络接口问题的beat来解决。

相关问题