kubernetes "健康处理器应包含必要的检查"测试在sig-api-machinery TC套件中失败，当kube-apiserver未直接暴露时,

myzjeezk  于 9个月前  发布在  Kubernetes
关注(0)|答案(4)|浏览(248)

发生了什么？
在执行CNTT测试用例时遇到了一个问题。
在我们的环境中，kube-apiserver没有直接暴露，可以通过集群内的nginx POD代理请求。
客户端(运行CNTT测试)=> POD(集群内运行，作为代理)=> kube-apiserver
kube-config已正确配置以访问该端点：

  1. clusters:
  2. - cluster:
  3.     server: https://x.y.z.73:8082/proxy/apiserver
  4. ...

执行sig_api_machinery测试用例时，"health handlers should contain necessary checks"测试用例失败。

  1. podman run -it --env-file ~/opnfv/env \
  2. -v ~/opnfv/ca.pem:/home/opnfv/functest/ca.pem:Z \
  3. -v ~/opnfv/config:/root/.kube/config:Z \
  4. -v ~/opnfv/repositories.yml:/home/opnfv/functest/repositories.yml:Z \
  5. -v ~/opnfv/results:/home/opnfv/functest/results:Z \
  6. opnfv/functest-kubernetes-smoke:v1.23 run_tests -t sig_api_machinery

结果：

  1. STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in health-9251
  2. STEP: Waiting for a default service account to be provisioned in namespace
  3. [It] should contain necessary checks
  4.  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/health_handlers.go:120
  5. STEP: /health
  6. Dec 15 13:54:27.529: FAIL: Unexpected error:
  7.    <*errors.errorString | 0xc0037fcd80>: {
  8.        s: "missing required /healthz?verbose=1 checks: map[[+]autoregister-completion ok:{} [+]etcd ok:{} [+]log ok:{} [+]ping ok:{} [+]poststarthook/apiservice-openapi-controller ok:{} [+]poststarthook/apiservice-registration-controller ok:{} [+]poststarthook/apiservice-status-available-controller ok:{} [+]poststarthook/bootstrap-controller ok:{} [+]poststarthook/crd-informer-synced ok:{} [+]poststarthook/generic-apiserver-start-informers ok:{} [+]poststarthook/kube-apiserver-autoregistration ok:{} [+]poststarthook/scheduling/bootstrap-system-priority-classes ok:{} [+]poststarthook/start-apiextensions-controllers ok:{} [+]poststarthook/start-apiextensions-informers ok:{} [+]poststarthook/start-cluster-authentication-info-controller ok:{} [+]poststarthook/start-kube-aggregator-informers ok:{} [+]poststarthook/start-kube-apiserver-admission-initializer ok:{}] in: ",
  9.    }
  10.    missing required /healthz?verbose=1 checks: map[[+]autoregister-completion ok:{} [+]etcd ok:{} [+]log ok:{} [+]ping ok:{} [+]poststarthook/apiservice-openapi-controller ok:{} [+]poststarthook/apiservice-registration-controller ok:{} [+]poststarthook/apiservice-status-available-controller ok:{} [+]poststarthook/bootstrap-controller ok:{} [+]poststarthook/crd-informer-synced ok:{} [+]poststarthook/generic-apiserver-start-informers ok:{} [+]poststarthook/kube-apiserver-autoregistration ok:{} [+]poststarthook/scheduling/bootstrap-system-priority-classes ok:{} [+]poststarthook/start-apiextensions-controllers ok:{} [+]poststarthook/start-apiextensions-informers ok:{} [+]poststarthook/start-cluster-authentication-info-controller ok:{} [+]poststarthook/start-kube-aggregator-informers ok:{} [+]poststarthook/start-kube-apiserver-admission-initializer ok:{}] in: 
  11. occurred
  13. Full Stack Trace
  14. k8s.io/kubernetes/test/e2e/apimachinery.glob..func14.1()
  15.        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/health_handlers.go:123 +0xc5
  16. k8s.io/kubernetes/test/e2e.RunE2ETests(0xc000d8e300)
  17.        _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c
  18. k8s.io/kubernetes/test/e2e.TestE2E(0xc000d8e300)
  19.        _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:144 +0x2b
  20. testing.tRunner(0xc000d8e300, 0x70f99e8)
  21.        /usr/local/go/src/testing/testing.go:1193 +0xef
  22. created by testing.(*T).Run
  23.        /usr/local/go/src/testing/testing.go:1238 +0x2b3
  24. [AfterEach] [sig-api-machinery] health handlers
  25.  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186
  26. STEP: Collecting events from namespace "health-9251".
  27. STEP: Found 0 events.
  29. ...
  31. • Failure [31.712 seconds]
  32. [sig-api-machinery] health handlers
  33. /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/framework.go:23
  34.  should contain necessary checks [It]
  35.  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/health_handlers.go:120
  37.  Dec 15 13:54:27.529: Unexpected error:
  38.      <*errors.errorString | 0xc0037fcd80>: {
  39.          s: "missing required /healthz?verbose=1 checks: map[[+]autoregister-completion ok:{} [+]etcd ok:{} [+]log ok:{} [+]ping ok:{} [+]poststarthook/apiservice-openapi-controller ok:{} [+]poststarthook/apiservice-registration-controller ok:{} [+]poststarthook/apiservice-status-available-controller ok:{} [+]poststarthook/bootstrap-controller ok:{} [+]poststarthook/crd-informer-synced ok:{} [+]poststarthook/generic-apiserver-start-informers ok:{} [+]poststarthook/kube-apiserver-autoregistration ok:{} [+]poststarthook/scheduling/bootstrap-system-priority-classes ok:{} [+]poststarthook/start-apiextensions-controllers ok:{} [+]poststarthook/start-apiextensions-informers ok:{} [+]poststarthook/start-cluster-authentication-info-controller ok:{} [+]poststarthook/start-kube-aggregator-informers ok:{} [+]poststarthook/start-kube-apiserver-admission-initializer ok:{}] in: ",
  40.      }
  41.      missing required /healthz?verbose=1 checks: map[[+]autoregister-completion ok:{} [+]etcd ok:{} [+]log ok:{} [+]ping ok:{} [+]poststarthook/apiservice-openapi-controller ok:{} [+]poststarthook/apiservice-registration-controller ok:{} [+]poststarthook/apiservice-status-available-controller ok:{} [+]poststarthook/bootstrap-controller ok:{} [+]poststarthook/crd-informer-synced ok:{} [+]poststarthook/generic-apiserver-start-informers ok:{} [+]poststarthook/kube-apiserver-autoregistration ok:{} [+]poststarthook/scheduling/bootstrap-system-priority-classes ok:{} [+]poststarthook/start-apiextensions-controllers ok:{} [+]poststarthook/start-apiextensions-informers ok:{} [+]poststarthook/start-cluster-authentication-info-controller ok:{} [+]poststarthook/start-kube-aggregator-informers ok:{} [+]poststarthook/start-kube-apiserver-admission-initializer ok:{}] in: 
  42.  occurred
  44.  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/health_handlers.go:123
  45. ------------------------------
  46. {"msg":"FAILED [sig-api-machinery] health handlers should contain necessary checks","total":1,"completed":0,"skipped":4622,"failed":1,"failures":["[sig-api-machinery] health handlers should contain necessary checks"]}

检查代码后，发现它与这里描述的问题相同：
#115200
因此，测试使用了使用RequestURI函数的testPath。因此，查询不使用kube-config中配置的完整路径，只保留IP:PORT和/health。这在我们的环境设置下当然会失败。
https://github.com/kubernetes/kubernetes/blob/release-1.23/test/e2e/apimachinery/health_handlers.go#L117
https://github.com/kubernetes/kubernetes/blob/release-1.23/staging/src/k8s.io/client-go/rest/request.go#L332

  1. func testPath(client clientset.Interface, path string, requiredChecks sets.String) error {
  2. 	var result restclient.Result
  3. 	err := wait.Poll(100*time.Millisecond, 30*time.Second, func() (bool, error) {
  4. 		result = client.CoreV1().RESTClient().Get().RequestURI(path).Do(context.TODO())
  5. 		status := 0
  6. 		result.StatusCode(&status)
  7. 		return status == 200, nil
  8. 	})
  9. 	if err != nil {
  10. 		return err
  11. 	}
  12. 	body, err := result.Raw()
  13. 	if err != nil {
  14. 		return err
  15. 	}
  16. 	checks := sets.NewString(strings.Split(string(body), "\n")...)
  17. 	if missing := requiredChecks.Difference(checks); missing.Len() > 0 {
  18. 		return fmt.Errorf("missing required %s checks: %v in: %s", path, missing, string(body))
  19. 	}
  20. 	return nil
  21. }
  23. var _ = SIGDescribe("health handlers", func() {
  24. 	f := framework.NewDefaultFramework("health")
  26. 	ginkgo.It("should contain necessary checks", func() {
  27. 		ginkgo.By("/health")
  28. 		err := testPath(f.ClientSet, "/healthz?verbose=1", requiredHealthzChecks)
  29. 		framework.ExpectNoError(err)
  31. 		ginkgo.By("/livez")
  32. 		err = testPath(f.ClientSet, "/livez?verbose=1", requiredLivezChecks)
  33. 		framework.ExpectNoError(err)
  35. 		ginkgo.By("/readyz")
  36. 		err = testPath(f.ClientSet, "/readyz?verbose=1", requiredReadyzChecks)
  37. 		framework.ExpectNoError(err)
  38. 	})
  39. })
  1. // RequestURI overwrites existing path and parameters with the value of the provided server relative
  2. // URI.
  3. func (r *Request) RequestURI(uri string) *Request {
  4. 	if r.err != nil {
  5. 		return r
  6. 	}
  7. 	locator, err := url.Parse(uri)
  8. 	if err != nil {
  9. 		r.err = err
  10. 		return r
  11. 	}
  12. 	r.pathPrefix = locator.Path
  13. 	if len(locator.Query()) > 0 {
  14. 		if r.params == nil {
  15. 			r.params = make(url.Values)
  16. 		}
  17. 		for k, v := range locator.Query() {
  18. 			r.params[k] = v
  19. 		}
  20. 	}
  21. 	return r
  22. }

我们期望发生什么？

测试用例应该成功执行。

我们如何尽可能精确地重现它？

您可以重新运行TC。

  1. podman run -it --env-file ~/opnfv/env -v ~/opnfv/ca.pem:/home/opnfv/functest/ca.pem:Z -v ~/opnfv/config:/root/.kube/config:Z -v ~/opnfv/repositories.yml:/home/opnfv/functest/repositories.yml:Z -v ~/opnfv/results:/home/opnfv/functest/results:Z opnfv/functest-kubernetes-smoke:v1.23 /bin/bash
  1. /usr/local/bin/e2e.test \
  2. --stderrthreshold=0 \
  3. --test.timeout=24h0m0s \
  4. --ginkgo.seed=1669123393 \
  5. --ginkgo.noColor \
  6. --ginkgo.slowSpecThreshold=5.00000 \
  7. -kubeconfig /root/.kube/config \
  8. -provider local \
  9. -report-dir /home/opnfv/functest/results/focus/ \
  10. -ginkgo.focus "health handlers should contain necessary checks" \
  11. -non-blocking-taints is_control,is_edge \
  12. -disable-log-dump false

我们需要了解的其他信息吗？

  • 无响应*

Kubernetes版本

  1. $ kubectl version
  2. Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.15", GitCommit:"b84cb8ab29366daa1bba65bc67f54de2f6c34848", GitTreeState:"clean", BuildDate:"2022-12-08T10:49:13Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
  3. Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.15", GitCommit:"b84cb8ab29366daa1bba65bc67f54de2f6c34848", GitTreeState:"clean", BuildDate:"2022-12-08T10:42:57Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}

云提供商

Openstack

OS版本

  1. # On Linux:
  2. $ cat /etc/os-release
  3. NAME="Rocky Linux"
  4. VERSION="8.6 (Green Obsidian)"
  5. ID="rocky"
  6. ID_LIKE="rhel centos fedora"
  7. VERSION_ID="8.6"
  8. PLATFORM_ID="platform:el8"
  9. PRETTY_NAME="Rocky Linux 8.6 (Green Obsidian)"
  10. ANSI_COLOR="0;32"
  11. CPE_NAME="cpe:/o:rocky:rocky:8:GA"
  12. HOME_URL="https://rockylinux.org/"
  13. BUG_REPORT_URL="https://bugs.rockylinux.org/"
  14. ROCKY_SUPPORT_PRODUCT="Rocky Linux"
  15. ROCKY_SUPPORT_PRODUCT_VERSION="8"
  16. REDHAT_SUPPORT_PRODUCT="Rocky Linux"
  17. REDHAT_SUPPORT_PRODUCT_VERSION="8"
  18. $ uname -a
  19. Linux cbis-sut1-control-01 4.18.0-372.9.1.el8.x86_64 #1 SMP Tue May 10 14:48:47 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

安装工具

容器运行时(CRI)和版本(如适用)

相关插件(CNI,CSI,...)和版本(如适用)

kubernetes

4条答案

按热度按时间
qq24tv8q

qq24tv8q1#

  1. /sig testing
  2. /sig api-machinery
  3. /area test
赞(0）分享  回复(0）举报  9个月前
sy5wg1nm

sy5wg1nm2#

  1. /assign @logicalhan
  2. /cc @fedebongio
  3. /triage accepted
赞(0）分享  回复(0）举报  9个月前
ev7lccsx

ev7lccsx3#

这个问题已经超过一年没有更新了，应该重新进行优先级评估。
你可以：

  • 确认这个问题仍然与 /triage accepted (仅组织成员)相关
  • 用 /close 关闭这个问题

有关优先级评估过程的更多详细信息，请参见 https://www.kubernetes.dev/docs/guide/issue-triage/
已接受移除优先级评估

赞(0）分享  回复(0）举报  9个月前
hof1towb

hof1towb4#

/triage accepted

赞(0）分享  回复(0）举报  9个月前
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com