发生了什么?
在重启kubelet后,kubelet开始输出与配额相关的错误,并且/etc/projects
和/etc/projid
中的条目总数减少了。
I0416 17:38:13.086019 157568 empty_dir.go:306] Set quota on /var/lib/kubelet/pods/e5149231-3a81-41e0-87be-579846f6caea/volumes/kubernetes.io~configmap/config failed assign quota FAILED exit status 1
I0416 17:38:13.086051 157568 operation_generator.go:838] "MountVolume.markVolumeErrorState leaving volume uncertain" volumeName=kubernetes.io/configmap/e5149231-3a81-41e0-87be-579846f6caea-config
E0416 17:38:13.086130 157568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5149231-3a81-41e0-87be-579846f6caea-config podName:e5149231-3a81-41e0-87be-579846f6caea nodeName:}" failed. No retries permitted until 2024-04-16 17:40:15.086106798 +0800 CST m=+376.261198810 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "config" (UniqueName: "kubernetes.io/configmap/e5149231-3a81-41e0-87be-579846f6caea-config") pod "xxx-yyy" (UID: "e5149231-3a81-41e0-87be-579846f6caea") : assign quota FAILED exit status 1
I0416 17:38:13.086156 157568 event.go:294] "Event occurred" object="xxx/yyy" fieldPath="" kind="Pod" apiVersion="v1" type="Warning" reason="FailedMount" message="MountVolume.SetUp failed for volume \"config\" : assign quota FAILED exit status 1"
相关问题: #115309
你期望发生什么?
我们如何尽可能精确地最小化地重现它?
- 启用并强制执行ext4 fs配额的
tune2fs -O project -Q prjquota /dev/xxx; mount -o prjquota ...
- 在kubelet配置中启用功能门
LocalStorageCapacityIsolationFSQuotaMonitoring
- 将此节点注册到k8s(启动kubelet):运行正常
- 重启kubelet:重现
我们需要知道其他任何信息吗?
- 无响应*
Kubernetes版本
$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.7", GitCommit:"84e1fc493a47446df2e155e70fca768d2653a398", GitTreeState:"clean", BuildDate:"2023-07-19T12:23:27Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.7", GitCommit:"84e1fc493a47446df2e155e70fca768d2653a398", GitTreeState:"clean", BuildDate:"2023-07-19T12:16:45Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
云提供商
IDC
操作系统版本
# On Linux:
$ cat /etc/os-release
NAME="AlmaLinux"
VERSION="9.3 (Shamrock Pampas Cat)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.3"
PLATFORM_ID="platform:el9"
PRETTY_NAME="AlmaLinux 9.3 (Shamrock Pampas Cat)"
ANSI_COLOR="0;34"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:almalinux:almalinux:9::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"
ALMALINUX_MANTISBT_PROJECT="AlmaLinux-9"
ALMALINUX_MANTISBT_PROJECT_VERSION="9.3"
REDHAT_SUPPORT_PRODUCT="AlmaLinux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.3"
$ uname -a
Linux xxx 5.14.0-362.24.2.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Mar 30 14:11:54 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux
安装工具
容器运行时(CRI)和版本(如适用)
containerd 1.6.31
相关插件(CNI,CSI等)和版本(如适用)
8条答案
按热度按时间ffdz8vbo1#
这个问题目前正在等待分类。
如果SIG或子项目确定这是一个相关的问题,他们将通过应用
triage/accepted
标签并提供进一步的指导来接受它。组织成员可以通过在评论中写入
/triage accepted
来添加triage/accepted
标签。有关使用PR评论与我互动的说明,请查看here。如果您对我的行为有任何问题或建议,请针对kubernetes/test-infra仓库提出一个问题。
eivgtgni2#
/sig node
nxagd54h3#
2ul0zpep4#
/sig storage
5uzkadbs5#
cc @pacoxu
bmvo0sr56#
CC @jingxu97
de90aj5v7#
Kubernetes项目目前缺乏足够的贡献者来充分应对所有问题。
此机器人根据以下规则对未分类的问题进行分级处理:
lifecycle/stale
应用后的90天不活动后,将应用lifecycle/stale
lifecycle/stale
应用后的30天不活动后,将应用lifecycle/rotten
lifecycle/rotten
应用后的30天不活动后,该问题将被关闭您可以:
/remove-lifecycle stale
/close
关闭此问题请将反馈发送至sig-contributor-experience@kubernetes/community。
/lifecycle stale
lstz6jyr8#
/remove-lifecycle stale