kubernetes "健康处理器应包含必要的检查"测试在sig-api-machinery TC套件中失败，当kube-apiserver未直接暴露时,

myzjeezk  于 5个月前  发布在  Kubernetes
关注(0)|答案(4)|浏览(118)

发生了什么？
在执行CNTT测试用例时遇到了一个问题。
在我们的环境中，kube-apiserver没有直接暴露，可以通过集群内的nginx POD代理请求。
客户端(运行CNTT测试)=> POD(集群内运行，作为代理)=> kube-apiserver
kube-config已正确配置以访问该端点：

clusters:
- cluster:
    server: https://x.y.z.73:8082/proxy/apiserver
...

执行sig_api_machinery测试用例时，"health handlers should contain necessary checks"测试用例失败。

podman run -it --env-file ~/opnfv/env \
-v ~/opnfv/ca.pem:/home/opnfv/functest/ca.pem:Z \
-v ~/opnfv/config:/root/.kube/config:Z \
-v ~/opnfv/repositories.yml:/home/opnfv/functest/repositories.yml:Z \
-v ~/opnfv/results:/home/opnfv/functest/results:Z \
opnfv/functest-kubernetes-smoke:v1.23 run_tests -t sig_api_machinery

结果：

STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in health-9251
STEP: Waiting for a default service account to be provisioned in namespace
[It] should contain necessary checks
 /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/health_handlers.go:120
STEP: /health
Dec 15 13:54:27.529: FAIL: Unexpected error:
   <*errors.errorString | 0xc0037fcd80>: {
       s: "missing required /healthz?verbose=1 checks: map[[+]autoregister-completion ok:{} [+]etcd ok:{} [+]log ok:{} [+]ping ok:{} [+]poststarthook/apiservice-openapi-controller ok:{} [+]poststarthook/apiservice-registration-controller ok:{} [+]poststarthook/apiservice-status-available-controller ok:{} [+]poststarthook/bootstrap-controller ok:{} [+]poststarthook/crd-informer-synced ok:{} [+]poststarthook/generic-apiserver-start-informers ok:{} [+]poststarthook/kube-apiserver-autoregistration ok:{} [+]poststarthook/scheduling/bootstrap-system-priority-classes ok:{} [+]poststarthook/start-apiextensions-controllers ok:{} [+]poststarthook/start-apiextensions-informers ok:{} [+]poststarthook/start-cluster-authentication-info-controller ok:{} [+]poststarthook/start-kube-aggregator-informers ok:{} [+]poststarthook/start-kube-apiserver-admission-initializer ok:{}] in: ",
   }
   missing required /healthz?verbose=1 checks: map[[+]autoregister-completion ok:{} [+]etcd ok:{} [+]log ok:{} [+]ping ok:{} [+]poststarthook/apiservice-openapi-controller ok:{} [+]poststarthook/apiservice-registration-controller ok:{} [+]poststarthook/apiservice-status-available-controller ok:{} [+]poststarthook/bootstrap-controller ok:{} [+]poststarthook/crd-informer-synced ok:{} [+]poststarthook/generic-apiserver-start-informers ok:{} [+]poststarthook/kube-apiserver-autoregistration ok:{} [+]poststarthook/scheduling/bootstrap-system-priority-classes ok:{} [+]poststarthook/start-apiextensions-controllers ok:{} [+]poststarthook/start-apiextensions-informers ok:{} [+]poststarthook/start-cluster-authentication-info-controller ok:{} [+]poststarthook/start-kube-aggregator-informers ok:{} [+]poststarthook/start-kube-apiserver-admission-initializer ok:{}] in: 
occurred

Full Stack Trace
k8s.io/kubernetes/test/e2e/apimachinery.glob..func14.1()
       /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/health_handlers.go:123 +0xc5
k8s.io/kubernetes/test/e2e.RunE2ETests(0xc000d8e300)
       _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c
k8s.io/kubernetes/test/e2e.TestE2E(0xc000d8e300)
       _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:144 +0x2b
testing.tRunner(0xc000d8e300, 0x70f99e8)
       /usr/local/go/src/testing/testing.go:1193 +0xef
created by testing.(*T).Run
       /usr/local/go/src/testing/testing.go:1238 +0x2b3
[AfterEach] [sig-api-machinery] health handlers
 /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186
STEP: Collecting events from namespace "health-9251".
STEP: Found 0 events.

...

• Failure [31.712 seconds]
[sig-api-machinery] health handlers
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/framework.go:23
 should contain necessary checks [It]
 /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/health_handlers.go:120

 Dec 15 13:54:27.529: Unexpected error:
     <*errors.errorString | 0xc0037fcd80>: {
         s: "missing required /healthz?verbose=1 checks: map[[+]autoregister-completion ok:{} [+]etcd ok:{} [+]log ok:{} [+]ping ok:{} [+]poststarthook/apiservice-openapi-controller ok:{} [+]poststarthook/apiservice-registration-controller ok:{} [+]poststarthook/apiservice-status-available-controller ok:{} [+]poststarthook/bootstrap-controller ok:{} [+]poststarthook/crd-informer-synced ok:{} [+]poststarthook/generic-apiserver-start-informers ok:{} [+]poststarthook/kube-apiserver-autoregistration ok:{} [+]poststarthook/scheduling/bootstrap-system-priority-classes ok:{} [+]poststarthook/start-apiextensions-controllers ok:{} [+]poststarthook/start-apiextensions-informers ok:{} [+]poststarthook/start-cluster-authentication-info-controller ok:{} [+]poststarthook/start-kube-aggregator-informers ok:{} [+]poststarthook/start-kube-apiserver-admission-initializer ok:{}] in: ",
     }
     missing required /healthz?verbose=1 checks: map[[+]autoregister-completion ok:{} [+]etcd ok:{} [+]log ok:{} [+]ping ok:{} [+]poststarthook/apiservice-openapi-controller ok:{} [+]poststarthook/apiservice-registration-controller ok:{} [+]poststarthook/apiservice-status-available-controller ok:{} [+]poststarthook/bootstrap-controller ok:{} [+]poststarthook/crd-informer-synced ok:{} [+]poststarthook/generic-apiserver-start-informers ok:{} [+]poststarthook/kube-apiserver-autoregistration ok:{} [+]poststarthook/scheduling/bootstrap-system-priority-classes ok:{} [+]poststarthook/start-apiextensions-controllers ok:{} [+]poststarthook/start-apiextensions-informers ok:{} [+]poststarthook/start-cluster-authentication-info-controller ok:{} [+]poststarthook/start-kube-aggregator-informers ok:{} [+]poststarthook/start-kube-apiserver-admission-initializer ok:{}] in: 
 occurred

 /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/health_handlers.go:123
------------------------------
{"msg":"FAILED [sig-api-machinery] health handlers should contain necessary checks","total":1,"completed":0,"skipped":4622,"failed":1,"failures":["[sig-api-machinery] health handlers should contain necessary checks"]}

检查代码后，发现它与这里描述的问题相同：
#115200
因此，测试使用了使用RequestURI函数的testPath。因此，查询不使用kube-config中配置的完整路径，只保留IP:PORT和/health。这在我们的环境设置下当然会失败。
https://github.com/kubernetes/kubernetes/blob/release-1.23/test/e2e/apimachinery/health_handlers.go#L117
https://github.com/kubernetes/kubernetes/blob/release-1.23/staging/src/k8s.io/client-go/rest/request.go#L332

func testPath(client clientset.Interface, path string, requiredChecks sets.String) error {
	var result restclient.Result
	err := wait.Poll(100*time.Millisecond, 30*time.Second, func() (bool, error) {
		result = client.CoreV1().RESTClient().Get().RequestURI(path).Do(context.TODO())
		status := 0
		result.StatusCode(&status)
		return status == 200, nil
	})
	if err != nil {
		return err
	}
	body, err := result.Raw()
	if err != nil {
		return err
	}
	checks := sets.NewString(strings.Split(string(body), "\n")...)
	if missing := requiredChecks.Difference(checks); missing.Len() > 0 {
		return fmt.Errorf("missing required %s checks: %v in: %s", path, missing, string(body))
	}
	return nil
}

var _ = SIGDescribe("health handlers", func() {
	f := framework.NewDefaultFramework("health")

	ginkgo.It("should contain necessary checks", func() {
		ginkgo.By("/health")
		err := testPath(f.ClientSet, "/healthz?verbose=1", requiredHealthzChecks)
		framework.ExpectNoError(err)

		ginkgo.By("/livez")
		err = testPath(f.ClientSet, "/livez?verbose=1", requiredLivezChecks)
		framework.ExpectNoError(err)

		ginkgo.By("/readyz")
		err = testPath(f.ClientSet, "/readyz?verbose=1", requiredReadyzChecks)
		framework.ExpectNoError(err)
	})
})
// RequestURI overwrites existing path and parameters with the value of the provided server relative
// URI.
func (r *Request) RequestURI(uri string) *Request {
	if r.err != nil {
		return r
	}
	locator, err := url.Parse(uri)
	if err != nil {
		r.err = err
		return r
	}
	r.pathPrefix = locator.Path
	if len(locator.Query()) > 0 {
		if r.params == nil {
			r.params = make(url.Values)
		}
		for k, v := range locator.Query() {
			r.params[k] = v
		}
	}
	return r
}

我们期望发生什么？

测试用例应该成功执行。

我们如何尽可能精确地重现它？

您可以重新运行TC。

podman run -it --env-file ~/opnfv/env -v ~/opnfv/ca.pem:/home/opnfv/functest/ca.pem:Z -v ~/opnfv/config:/root/.kube/config:Z -v ~/opnfv/repositories.yml:/home/opnfv/functest/repositories.yml:Z -v ~/opnfv/results:/home/opnfv/functest/results:Z opnfv/functest-kubernetes-smoke:v1.23 /bin/bash
/usr/local/bin/e2e.test \
--stderrthreshold=0 \
--test.timeout=24h0m0s \
--ginkgo.seed=1669123393 \
--ginkgo.noColor \
--ginkgo.slowSpecThreshold=5.00000 \
-kubeconfig /root/.kube/config \
-provider local \
-report-dir /home/opnfv/functest/results/focus/ \
-ginkgo.focus "health handlers should contain necessary checks" \
-non-blocking-taints is_control,is_edge \
-disable-log-dump false

我们需要了解的其他信息吗？

  • 无响应*

Kubernetes版本

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.15", GitCommit:"b84cb8ab29366daa1bba65bc67f54de2f6c34848", GitTreeState:"clean", BuildDate:"2022-12-08T10:49:13Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.15", GitCommit:"b84cb8ab29366daa1bba65bc67f54de2f6c34848", GitTreeState:"clean", BuildDate:"2022-12-08T10:42:57Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}

云提供商

Openstack

OS版本

# On Linux:
$ cat /etc/os-release
NAME="Rocky Linux"
VERSION="8.6 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.6 (Green Obsidian)"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky Linux"
ROCKY_SUPPORT_PRODUCT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8"
$ uname -a
Linux cbis-sut1-control-01 4.18.0-372.9.1.el8.x86_64 #1 SMP Tue May 10 14:48:47 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

安装工具

容器运行时(CRI)和版本(如适用)

相关插件(CNI,CSI,...)和版本(如适用)

kubernetes

4条答案

按热度按时间
qq24tv8q

qq24tv8q1#

/sig testing
/sig api-machinery
/area test
赞(0）分享  回复(0）举报  5个月前
sy5wg1nm

sy5wg1nm2#

/assign @logicalhan
/cc @fedebongio
/triage accepted
赞(0）分享  回复(0）举报  5个月前
ev7lccsx

ev7lccsx3#

这个问题已经超过一年没有更新了，应该重新进行优先级评估。
你可以：

  • 确认这个问题仍然与 /triage accepted (仅组织成员)相关
  • 用 /close 关闭这个问题

有关优先级评估过程的更多详细信息，请参见 https://www.kubernetes.dev/docs/guide/issue-triage/
已接受移除优先级评估

赞(0）分享  回复(0）举报  5个月前
hof1towb

hof1towb4#

/triage accepted

赞(0）分享  回复(0）举报  5个月前
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com