kubernetes 通过负载平衡器将外部流量路由到入口还是仅通过aks上的入口？

我有一个配置了LoadBalancer的AKS集群（在https://learn.microsoft.com/en-us/azure/aks/internal-lb之后），以便它从PublicIP（都是用Terraform提供的）获得IP，并将Helm部署的集群入口作为目标。

resource "kubernetes_service" "server-loadbalacer" {
  metadata {
    name = "server-loadbalacer-svc"
    annotations = {
      "service.beta.kubernetes.io/azure-load-balancer-resource-group" = "fixit-resource-group"
    }
  }
  spec {
    type = "LoadBalancer"
    load_balancer_ip = var.public_ip_address
    selector = {
      name = "ingress-service"
    }
    port {
      name = "server-port"
      protocol = "TCP"
      port = 8080
    }
  }
}

然后使用Helm部署了一个侦听端口3000的Node.js服务器、一个MongoDB副本集和一个Neo4集群，并为服务器设置了一个在端口3000上接收和以端口3000为目标的服务。

apiVersion: v1
kind: Service
metadata:
  name: server-clusterip-service
spec:
  type: ClusterIP
  selector:
    app: fixit-server-pod
  ports:
    - name: server-clusterip-service
      protocol: TCP
      port: 3000 # service port
      targetPort: 3000 # por on whic the app is listening to

然后入口将流量重定向到正确的服务，例如服务器

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    kubernetes.io/ingress.class: nginx
  labels:
    name: ingress-service
spec:
  rules:
  - host: fixit.westeurope.cloudapp.azure.com #dns from Azure PublicIP
    http:
      paths:
        - path: '/server/*'
          pathType: Prefix
          backend:
            service:
              name: server-clusterip-service
              port:
                number: 3000
        - path: '/neo4j/*'
          pathType: Prefix
          backend: 
            service:
              name: fixit-cluster
              port:
                number: 7687
                number: 7474
                number: 7473
        - path: '/neo4j-admin/*'
          pathType: Prefix
          backend:
            service: 
              name: fixit-cluster-admin
              port:
                number: 6362
                number: 7687
                number: 7474
                number: 7473

我希望转到http://fixit.westeurope.cloudapp.azure.com:8080/server/api并看到这样的消息：服务器对端点/api做出响应，但由于浏览器超时而失败。

vincenzocalia@vincenzos-MacBook-Air helm_charts % kubectl get pod
NAME                                           READY   STATUS    RESTARTS   AGE
fixit-cluster-0                                1/1     Running   0          27m
fixit-server-868f657b64-hvmxq                  1/1     Running   0          27m
mongo-rs-0                                     2/2     Running   0          27m
mongodb-kubernetes-operator-7c5666c957-sscsf   1/1     Running   0          4h35m
vincenzocalia@vincenzos-MacBook-Air helm_charts % kubectl get svc  
NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)                               AGE
fixit-cluster              ClusterIP      10.0.230.247   <none>         7687/TCP,7474/TCP,7473/TCP            27m
fixit-cluster-admin        ClusterIP      10.0.132.24    <none>         6362/TCP,7687/TCP,7474/TCP,7473/TCP   27m
kubernetes                 ClusterIP      10.0.0.1       <none>         443/TCP                               4h44m
mongo-rs-svc               ClusterIP      None           <none>         27017/TCP                             27m
server-clusterip-service   ClusterIP      10.0.242.65    <none>         3000/TCP                              27m
server-loadbalacer-svc     LoadBalancer   10.0.149.160   52.174.18.27   8080:32660/TCP                        4h41m

入口部署为

vincenzocalia@vincenzos-MacBook-Air helm_charts % kubectl describe ingress ingress-service
Name:             ingress-service
Labels:           app.kubernetes.io/managed-by=Helm
                  name=ingress-service
Namespace:        default
Address:          
Ingress Class:    <none>
Default backend:  <default>
Rules:
  Host                                 Path  Backends
  ----                                 ----  --------
  fixit.westeurope.cloudapp.azure.com  
                                       /server/*        server-clusterip-service:3000 (<none>)
                                       /neo4j/*         fixit-cluster:7473 (<none>)
                                       /neo4j-admin/*   fixit-cluster-admin:7473 (<none>)
Annotations:                           kubernetes.io/ingress.class: nginx
                                       meta.helm.sh/release-name: fixit-cluster
                                       meta.helm.sh/release-namespace: default
Events:                                <none>

所述服务器服务

vincenzocalia@vincenzos-MacBook-Air helm_charts % kubectl describe svc server-clusterip-service
Name:              server-clusterip-service
Namespace:         default
Labels:            app.kubernetes.io/managed-by=Helm
Annotations:       meta.helm.sh/release-name: fixit-cluster
                   meta.helm.sh/release-namespace: default
Selector:          app=fixit-server-pod
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.0.160.206
IPs:               10.0.160.206
Port:              server-clusterip-service  3000/TCP
TargetPort:        3000/TCP
Endpoints:         10.244.0.15:3000
Session Affinity:  None
Events:            <none>

我试着设置路径有和没有/*，但它不会在任何情况下连接。这种设置甚至是正确的方式来路由外部流量到集群或我应该只使用入口？我看到这个设置已作为解决方案提供（第1个答案）对于这个问题Kubernetes Load balancer without Label Selector和dough看起来我们的情况是一样的，我在AKS上，和Azure文档https://learn.microsoft.com/en-us/azure/aks/ingress-basic?tabs=azure-cli让我对我目前的设置产生了怀疑。如果这个设置不是毫无意义的，你能发现我设置错了什么吗？非常感谢你的帮助。
更新
如此处所述https://learnk8s.io/terraform-aks群集创建中的选项http_application_routing_enabled = true安装插件

vincenzocalia@vincenzos-MacBook-Air helm_charts % kubectl get pods -n kube-system | grep addon
addon-http-application-routing-external-dns-5d48bdffc6-q98nx      1/1     Running   0          26m
addon-http-application-routing-nginx-ingress-controller-5bcrf87   1/1     Running   0          26m

因此入口服务应该在其注解中指向该控制器，而不是指定主机，所以我将入口服务更改为

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    # kubernetes.io/ingress.class: nginx
    kubernetes.io/ingress.class: addon-http-application-routing

    # nginx.ingress.kubernetes.io/rewrite-target: /

  labels:
    name: ingress-service
spec:
  rules:
  # - host: fixit.westeurope.cloudapp.azure.com #server.com 
    - http:
       paths:
         - path: '/server/*' # service
        #  - path: '/server' # service doesn't get a IPaddress
         # - path: '/*'
         # - path: '/'
           pathType: Prefix
           backend:
             service:
               name: server-clusterip-service
               port:
                 number: 3000
        # - path: '/neo4j/*'
        #   pathType: Prefix
        #   backend: 
        #     service:
        #       name: fixit-cluster
        #       port:
        #         number: 7687
        #         number: 7474
        #         number: 7473
        # - path: '/neo4j-admin/*'
        #   pathType: Prefix
        #   backend:
        #     service: 
        #       name: fixit-cluster-admin
        #       port:
        #         number: 6362
        #         number: 7687
        #         number: 7474
        #         number: 7473

它的输出现在是

vincenzocalia@vincenzos-MacBook-Air helm_charts % kubectl get ingress
NAME              CLASS    HOSTS   ADDRESS          PORTS   AGE
ingress-service   <none>   *       108.143.71.248   80      7s
vincenzocalia@vincenzos-MacBook-Air helm_charts % kubectl describe ingress ingress-service 
Name:             ingress-service
Labels:           app.kubernetes.io/managed-by=Helm
                  name=ingress-service
Namespace:        default
Address:          108.143.71.248
Ingress Class:    <none>
Default backend:  <default>
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /server/*   server-clusterip-service:3000 (10.244.0.21:3000)
Annotations:  kubernetes.io/ingress.class: addon-http-application-routing
              meta.helm.sh/release-name: fixit-cluster
              meta.helm.sh/release-namespace: default
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Normal  Sync    20s (x2 over 27s)  nginx-ingress-controller  Scheduled for sync

现在转到浏览器中的http://108.143.71.248/server/api将显示Nginx 404页面。

我终于发现了问题。这是我的设置。我使用的是默认的入口控制器和负载平衡器，当你设置集群创建选项http_application_routing_enabled = true时，文档不鼓励在生产https://learn.microsoft.com/en-us/azure/aks/http-application-routing中使用。所以正确的实现是安装一个入口控制器https://learn.microsoft.com/en-us/azure/aks/ingress-basic?tabs=azure-cli，它连接内部负载平衡器，所以没有必要创建一个。现在，入口控制器将接受负载平衡器的IP地址，但您必须在node resource group中创建PublicIP，因为将在node resource group中查找它，而不是在resource group中查找。请在www.example.com中检查两者之间的差异https://learn.microsoft.com/en-us/azure/aks/faq#why-are-two-resource-groups-created-with-aks。
因此，现在的工作配置为：

主电源

terraform {
  required_version = ">=1.1.0"
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
       version = "~> 3.0.2"
    }
  }
}

provider "azurerm" {
  features {
    resource_group {
      prevent_deletion_if_contains_resources = false
    }
  }
  subscription_id   = var.azure_subscription_id
  tenant_id         = var.azure_subscription_tenant_id
  client_id         = var.service_principal_appid
  client_secret     = var.service_principal_password
}

provider "kubernetes" {
  host = "${module.cluster.host}"
  client_certificate = "${base64decode(module.cluster.client_certificate)}"
  client_key = "${base64decode(module.cluster.client_key)}"
  cluster_ca_certificate = "${base64decode(module.cluster.cluster_ca_certificate)}"
}

provider "helm" {
  kubernetes {
    host                   = "${module.cluster.host}"
    client_certificate     = "${base64decode(module.cluster.client_certificate)}"
    client_key             = "${base64decode(module.cluster.client_key)}"
    cluster_ca_certificate = "${base64decode(module.cluster.cluster_ca_certificate)}"
  }
}


module "cluster" {
  source = "./modules/cluster"
  location = var.location
  vm_size = var.vm_size
  resource_group_name = var.resource_group_name
  node_resource_group_name = var.node_resource_group_name
  kubernetes_version = var.kubernetes_version
  ssh_key = var.ssh_key
  sp_client_id = var.service_principal_appid
  sp_client_secret = var.service_principal_password
}



module "ingress-controller" {
  source = "./modules/ingress-controller"
  public_ip_address = module.cluster.public_ip_address
  depends_on = [
    module.cluster.public_ip_address
  ]
}

簇

resource "azurerm_resource_group" "resource_group" {
  name     = var.resource_group_name
  location = var.location
    tags = {
    Environment = "test"
    Team = "DevOps"
  }
}
resource "azurerm_kubernetes_cluster" "server_cluster" {
  name                = "server_cluster"
  ### choose the resource goup to use for the cluster
  location            = azurerm_resource_group.resource_group.location
  resource_group_name = azurerm_resource_group.resource_group.name
  ### decide the name of the cluster "node" resource group, if unset will be named automatically 
  node_resource_group = var.node_resource_group_name
  dns_prefix          = "fixit"
  kubernetes_version = var.kubernetes_version
  # sku_tier = "Paid"

  default_node_pool {
    name       = "default"
    node_count = 1
    min_count = 1
    max_count = 3
    vm_size = var.vm_size

    type = "VirtualMachineScaleSets"
    enable_auto_scaling = true
    enable_host_encryption = false
    # os_disk_size_gb = 30
  }

  service_principal {
    client_id = var.sp_client_id
    client_secret = var.sp_client_secret
  }

  tags = {
    Environment = "Production"
  }

  linux_profile {
    admin_username = "azureuser"
    ssh_key {
        key_data = var.ssh_key
    }
  }
  network_profile {
      network_plugin = "kubenet"
      load_balancer_sku = "basic" 
    
  }
  http_application_routing_enabled = false
  depends_on = [
    azurerm_resource_group.resource_group
  ]
}

resource "azurerm_public_ip" "public-ip" {
  name                = "fixit-public-ip"
  location            = var.location
  # resource_group_name = var.resource_group_name
  resource_group_name = var.node_resource_group_name
  allocation_method   = "Static"
  domain_name_label = "fixit"
  # sku = "Standard"

depends_on = [
  azurerm_kubernetes_cluster.server_cluster
]
}

入口控制器

resource "helm_release" "nginx" {
  name      = "ingress-nginx"
  repository = "ingress-nginx"
  chart     = "ingress-nginx/ingress-nginx"
  namespace = "default"

  set {
    name  = "controller.service.externalTrafficPolicy"
    value = "Local"
  }

  set {
    name = "controller.service.annotations.service.beta.kubernetes.io/azure-load-balancer-internal"
    value = "true"
  }

  set {
    name  = "controller.service.loadBalancerIP"
    value = var.public_ip_address
  }

  set {
    name  = "controller.service.annotations.service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path"
    value = "/healthz"
  }
}

入口服务

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-service
  # namespace: default
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$2$3$4
spec:
  ingressClassName: nginx
  rules:
    # - host: fixit.westeurope.cloudapp.azure.com #dns from Azure PublicIP

### Node.js server
  - http:
      paths:
      - path: /(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: server-clusterip-service
            port:
              number: 80 

  - http:
      paths:
      - path: /server(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: server-clusterip-service
            port:
              number: 80
...

other services omitted

希望这能帮助你把设置弄对，干杯。

kubernetes 通过负载平衡器将外部流量路由到入口还是仅通过aks上的入口？

1条答案

主电源

簇

入口控制器

入口服务

相关问题

热门标签

最新问答