kubernetes Cilium 1.12.1在某些节点上无法正常工作

pb3s4cty  于 2022-11-02  发布在  Kubernetes
关注(0)|答案(1)|浏览(228)

大家好，我有一个11节点的Kubernetes集群，cilium 1.12.1 kubeProxyReplacement=strict构建在我们的数据中心的裸机上，但是其中4个节点（node 5-node 8）上的pod在与其他pod或不在同一节点上的服务通信时出现问题，其他7个节点没有问题。我可以ping到其他pod的IP，但是当telnet端口时，包似乎从未到达。
所有11个节点都安装了相同版本的操作系统、相同的内核，并且使用Kubespray部署集群，我确保这11个节点尽可能具有相同的软件环境（我不确定这是否与硬件有关，但4个有问题的节点是千兆网卡服务器，其他节点都是10千兆网卡）。
这是节点列表：

❯ kubectl get nodes -o wide
NAME                          STATUS   ROLES           AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION    CONTAINER-RUNTIME
master01   Ready    control-plane   39h   v1.24.4   10.252.55.22     <none>        CentOS Linux 7 (Core)   5.10.0-1.0.0.17   containerd://1.6.8
master02   Ready    control-plane   39h   v1.24.4   10.252.54.44     <none>        CentOS Linux 7 (Core)   5.10.0-1.0.0.17   containerd://1.6.8
master03   Ready    control-plane   39h   v1.24.4   10.252.55.39     <none>        CentOS Linux 7 (Core)   5.10.0-1.0.0.17   containerd://1.6.8
node05     Ready    <none>          39h   v1.24.4   10.252.34.27     <none>        CentOS Linux 7 (Core)   5.10.0-1.0.0.17   containerd://1.6.8
node06     Ready    <none>          39h   v1.24.4   10.252.33.44     <none>        CentOS Linux 7 (Core)   5.10.0-1.0.0.17   containerd://1.6.8
node07     Ready    <none>          39h   v1.24.4   10.252.33.52     <none>        CentOS Linux 7 (Core)   5.10.0-1.0.0.17   containerd://1.6.8
node08     Ready    <none>          39h   v1.24.4   10.252.33.45     <none>        CentOS Linux 7 (Core)   5.10.0-1.0.0.17   containerd://1.6.8
node01     Ready    <none>          39h   v1.24.4   10.252.144.206   <none>        CentOS Linux 7 (Core)   5.10.0-1.0.0.17   containerd://1.6.8
node02     Ready    <none>          39h   v1.24.4   10.252.145.13    <none>        CentOS Linux 7 (Core)   5.10.0-1.0.0.17   containerd://1.6.8
node03     Ready    <none>          39h   v1.24.4   10.252.145.163   <none>        CentOS Linux 7 (Core)   5.10.0-1.0.0.17   containerd://1.6.8
node04     Ready    <none>          39h   v1.24.4   10.252.145.226   <none>        CentOS Linux 7 (Core)   5.10.0-1.0.0.17   containerd://1.6.8

这是node 5上的pod与master 01上运行的nginx pod通信时发生的情况：


# ping works fine

bash-5.1# ping 10.233.64.103
PING 10.233.64.103 (10.233.64.103) 56(84) bytes of data.
64 bytes from 10.233.64.103: icmp_seq=1 ttl=63 time=0.214 ms
64 bytes from 10.233.64.103: icmp_seq=2 ttl=63 time=0.148 ms
--- 10.233.64.103 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1026ms
rtt min/avg/max/mdev = 0.148/0.181/0.214/0.033 ms

# curl not working

bash-5.1# curl 10.233.64.103
curl: (28) Failed to connect to 10.233.64.103 port 80 after 3069 ms: Operation timed out

# hubble observe logs(hubble observe --to-ip 10.233.64.103 -f):

Sep  6 03:15:16.100: cilium-test/testubuntu-g2gv6 (ID:9268) -> cilium-test/nginx-deployment-bpvnx (ID:4221) to-overlay FORWARDED (ICMPv4 EchoRequest)
Sep  6 03:15:16.100: cilium-test/testubuntu-g2gv6 (ID:9268) -> cilium-test/nginx-deployment-bpvnx (ID:4221) to-endpoint FORWARDED (ICMPv4 EchoRequest)
Sep  6 03:15:22.026: cilium-test/testubuntu-g2gv6:33722 (ID:9268) -> cilium-test/nginx-deployment-bpvnx:80 (ID:4221) to-overlay FORWARDED (TCP Flags: SYN)

这是node 4上的pod在与同一nginx pod通信时所发生的情况：


# ping works fine

bash-5.1# ping 10.233.64.103
PING 10.233.64.103 (10.233.64.103) 56(84) bytes of data.
64 bytes from 10.233.64.103: icmp_seq=1 ttl=63 time=2.33 ms
64 bytes from 10.233.64.103: icmp_seq=2 ttl=63 time=2.30 ms

# curl works fine as well

bash-5.1# curl  10.233.64.103
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

# hubble observe logs(hubble observe --to-ip 10.233.64.103 -f):

Sep  6 03:16:24.808: cilium-test/testubuntu-wcwfg (ID:9268) -> cilium-test/nginx-deployment-bpvnx (ID:4221) to-overlay FORWARDED (ICMPv4 EchoRequest)
Sep  6 03:16:24.810: cilium-test/testubuntu-wcwfg (ID:9268) -> cilium-test/nginx-deployment-bpvnx (ID:4221) to-endpoint FORWARDED (ICMPv4 EchoRequest)
Sep  6 03:16:27.043: cilium-test/testubuntu-wcwfg:57802 (ID:9268) -> cilium-test/nginx-deployment-bpvnx:80 (ID:4221) to-overlay FORWARDED (TCP Flags: SYN)
Sep  6 03:16:27.045: cilium-test/testubuntu-wcwfg:57802 (ID:9268) -> cilium-test/nginx-deployment-bpvnx:80 (ID:4221) to-endpoint FORWARDED (TCP Flags: SYN)
Sep  6 03:16:27.045: cilium-test/testubuntu-wcwfg:57802 (ID:9268) -> cilium-test/nginx-deployment-bpvnx:80 (ID:4221) to-overlay FORWARDED (TCP Flags: ACK)
Sep  6 03:16:27.045: cilium-test/testubuntu-wcwfg:57802 (ID:9268) -> cilium-test/nginx-deployment-bpvnx:80 (ID:4221) to-overlay FORWARDED (TCP Flags: ACK, PSH)
Sep  6 03:16:27.047: cilium-test/testubuntu-wcwfg:57802 (ID:9268) -> cilium-test/nginx-deployment-bpvnx:80 (ID:4221) to-endpoint FORWARDED (TCP Flags: ACK)
Sep  6 03:16:27.047: cilium-test/testubuntu-wcwfg:57802 (ID:9268) -> cilium-test/nginx-deployment-bpvnx:80 (ID:4221) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Sep  6 03:16:27.048: cilium-test/testubuntu-wcwfg:57802 (ID:9268) -> cilium-test/nginx-deployment-bpvnx:80 (ID:4221) to-overlay FORWARDED (TCP Flags: ACK, FIN)
Sep  6 03:16:27.050: cilium-test/testubuntu-wcwfg:57802 (ID:9268) -> cilium-test/nginx-deployment-bpvnx:80 (ID:4221) to-endpoint FORWARDED (TCP Flags: ACK, FIN)
Sep  6 03:16:27.050: cilium-test/testubuntu-wcwfg:57802 (ID:9268) -> cilium-test/nginx-deployment-bpvnx:80 (ID:4221) to-overlay FORWARDED (TCP Flags: ACK)
Sep  6 03:16:27.051: cilium-test/testubuntu-wcwfg:57802 (ID:9268) -> cilium-test/nginx-deployment-bpvnx:80 (ID:4221) to-endpoint FORWARDED (TCP Flags: ACK)

这是cilium-health status，还显示了4个节点上的端口连接问题：

❯ kubectl exec -it -n kube-system ds/cilium -- cilium-health status
Defaulted container "cilium-agent" out of: cilium-agent, mount-cgroup (init), clean-cilium-state (init)
Probe time:   2022-09-06T03:10:24Z
Nodes:
  node01 (localhost):
    Host connectivity to 10.252.144.206:
      ICMP to stack:   OK, RTT=341.295µs
      HTTP to agent:   OK, RTT=100.729µs
    Endpoint connectivity to 10.233.67.53:
      ICMP to stack:   OK, RTT=334.224µs
      HTTP to agent:   OK, RTT=163.289µs
  master01:
    Host connectivity to 10.252.55.22:
      ICMP to stack:   OK, RTT=1.994728ms
      HTTP to agent:   OK, RTT=1.610932ms
    Endpoint connectivity to 10.233.64.235:
      ICMP to stack:   OK, RTT=2.100332ms
      HTTP to agent:   OK, RTT=2.489126ms
  master02:
    Host connectivity to 10.252.54.44:
      ICMP to stack:   OK, RTT=2.33033ms
      HTTP to agent:   OK, RTT=2.34166ms
    Endpoint connectivity to 10.233.65.225:
      ICMP to stack:   OK, RTT=2.101561ms
      HTTP to agent:   OK, RTT=2.067012ms
  master03:
    Host connectivity to 10.252.55.39:
      ICMP to stack:   OK, RTT=1.688641ms
      HTTP to agent:   OK, RTT=1.593428ms
    Endpoint connectivity to 10.233.66.74:
      ICMP to stack:   OK, RTT=2.210915ms
      HTTP to agent:   OK, RTT=1.725555ms
  node05:
    Host connectivity to 10.252.34.27:
      ICMP to stack:   OK, RTT=2.383001ms
      HTTP to agent:   OK, RTT=2.48362ms
    Endpoint connectivity to 10.233.70.87:
      ICMP to stack:   OK, RTT=2.194843ms
      HTTP to agent:   Get "http://10.233.70.87:4240/hello": dial tcp 10.233.70.87:4240: connect: connection timed out
  node06:
    Host connectivity to 10.252.33.44:
      ICMP to stack:   OK, RTT=2.091932ms
      HTTP to agent:   OK, RTT=1.724729ms
    Endpoint connectivity to 10.233.71.119:
      ICMP to stack:   OK, RTT=1.984056ms
      HTTP to agent:   Get "http://10.233.71.119:4240/hello": dial tcp 10.233.71.119:4240: connect: connection timed out
  node07:
    Host connectivity to 10.252.33.52:
      ICMP to stack:   OK, RTT=2.055482ms
      HTTP to agent:   OK, RTT=2.037437ms
    Endpoint connectivity to 10.233.72.47:
      ICMP to stack:   OK, RTT=1.853614ms
      HTTP to agent:   Get "http://10.233.72.47:4240/hello": dial tcp 10.233.72.47:4240: connect: connection timed out
  node08:
    Host connectivity to 10.252.33.45:
      ICMP to stack:   OK, RTT=2.461315ms
      HTTP to agent:   OK, RTT=2.369003ms
    Endpoint connectivity to 10.233.74.247:
      ICMP to stack:   OK, RTT=2.097029ms
      HTTP to agent:   Get "http://10.233.74.247:4240/hello": dial tcp 10.233.74.247:4240: connect: connection timed out
  node02:
    Host connectivity to 10.252.145.13:
      ICMP to stack:   OK, RTT=372.787µs
      HTTP to agent:   OK, RTT=168.915µs
    Endpoint connectivity to 10.233.73.98:
      ICMP to stack:   OK, RTT=360.354µs
      HTTP to agent:   OK, RTT=287.224µs
  node03:
    Host connectivity to 10.252.145.163:
      ICMP to stack:   OK, RTT=363.072µs
      HTTP to agent:   OK, RTT=216.652µs
    Endpoint connectivity to 10.233.68.73:
      ICMP to stack:   OK, RTT=312.153µs
      HTTP to agent:   OK, RTT=304.981µs
  node04:
    Host connectivity to 10.252.145.226:
      ICMP to stack:   OK, RTT=375.121µs
      HTTP to agent:   OK, RTT=185.484µs
    Endpoint connectivity to 10.233.69.140:
      ICMP to stack:   OK, RTT=403.752µs
      HTTP to agent:   OK, RTT=277.517µs

对于我应该从哪里开始进行故障诊断，有什么建议吗？

kubernetes

1条答案

按热度按时间
blmhpbnm

blmhpbnm1#

从1.12版本开始，他们对路由做了很大的修改。尝试启用传统路由。
在helm_values.yaml中（如果您使用helm进行部署），您应该添加：

bpf:
  hostLegacyRouting: true

它配置直接路由模式是通过主机堆栈（true）还是直接且更有效地从BPF（false）路由流量（如果内核支持它）。后者意味着它还将绕过主机名称空间中的netfilter。
关于BPF的详细信息可以在官方文档中找到，注意节点操作系统与BPF的兼容性

赞(0）分享  回复(0）举报  2022-11-02
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com