简介
本文档介绍恢复kube-apiserver pod持续重启的解决方案。
先决条件
要求
思科建议您了解以下主题：
- 多克斯和库伯内特斯
- 思科用户微服务基础设施(SMI)超云核心通用执行环境(CEE)
使用的组件
本文档中的信息基于Kubernetes v1.21.0版。
本文档中的信息都是基于特定实验室环境中的设备编写的。本文档中使用的所有设备最初均采用原始（默认）配置。如果您的网络处于活动状态，请确保您了解所有命令的潜在影响。
什么是kube-apiserver?
a.验证用户
b.验证请求
c.检索数据
d.更新ETCD
e.调度程序
f.库贝莱
- 其他组件（如调度程序、kube-controller-manager和kubelet）使用API服务器在集群中各自区域执行更新。
问题
持续观察kube-apiserver-smf-data-master-3重新启动。在这种情况下，执行kubectl CLI kubectl get pods -A -o wide | grep apiserver以确定问题：
cloud-user@smf-data-master-1:~$ kubectl get pods -A -o wide | grep apiserver
kube-system kube-apiserver-smf-data-master-1 1/1 Running 4 68d 10.192.1.22 smf-data-master-1 <none> <none>
kube-system kube-apiserver-smf-data-master-2 1/1 Running 4 68d 10.192.1.23 smf-data-master-2 <none> <none>
kube-system kube-apiserver-smf-data-master-3 0/1 Running 2 68d 10.192.1.24 smf-data-master-3 <none> <none>
cloud-user@smf-data-master-1:~$
在kubectl日志<kube-apiserver_pod_name> -n kube-system中观察到以下错误：
cloud-user@smf-data-master-1:~$ kubectl logs kube-apiserver-smf-data-master-3 -n kube-system
E1116 20:09:52.635602 1 cacher.go:419] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/cee-dnceed21/alert-logger-sa-token-dzhkb": invalid padding on input; reinitializing...
E1116 20:09:53.691253 1 cacher.go:419] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/cee-dnceed21/alert-logger-sa-token-dzhkb": invalid padding on input; reinitializing...
E1116 20:09:54.751145 1 cacher.go:419] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/cee-dnceed21/alert-logger-sa-token-dzhkb": invalid padding on input; reinitializing...
E1116 20:09:55.808782 1 cacher.go:419] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/cee-dnceed21/alert-logger-sa-token-dzhkb": invalid padding on input; reinitializing...
E1116 20:09:56.865492 1 cacher.go:419] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/cee-dnceed21/alert-logger-sa-token-dzhkb": invalid padding on input; reinitializing...
E1116 20:09:57.906426 1 cacher.go:419] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/cee-dnceed21/alert-logger-sa-token-dzhkb": invalid padding on input; reinitializing...
E1116 20:09:58.963801 1 cacher.go:419] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/cee-dnceed21/alert-logger-sa-token-dzhkb": invalid padding on input; reinitializing...
E1116 20:10:00.027583 1 cacher.go:419] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/cee-dnceed21/alert-logger-sa-token-dzhkb": invalid padding on input; reinitializing...
E1116 20:10:01.084615 1 cacher.go:419] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/cee-dnceed21/alert-logger-sa-token-dzhkb": invalid padding on input; reinitializing...
E1116 20:10:02.206947 1 cacher.go:419] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/cee-dnceed21/alert-logger-sa-token-dzhkb": invalid padding on input; reinitializing...
E1116 20:10:03.256261 1 cacher.go:419] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/cee-dnceed21/alert-logger-sa-token-dzhkb": invalid padding on input; reinitializing...
E1116 20:10:04.313860 1 cacher.go:419] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/cee-dnceed21/alert-logger-sa-token-dzhkb": invalid padding on input; reinitializing...
E1116 20:10:05.363353 1 cacher.go:419] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/cee-dnceed21/alert-logger-sa-token-dzhkb": invalid padding on input; reinitializing...
要恢复，必须尝试使用CLI kubectl delete pod <kube-apiserver_pod_name> -n kube-system重新启动kube-apiserver pod，但它没有帮助。
根本原因分析
进一步分析发现，在kube-apiserver持续重启的master-3和其他master节点之间的secret 值差异导致了此问题。
From Master-1:
cloud-user@smf-data-master-1:~$ cat /data/kubernetes/secrets.conf
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: BG5hleucjlD5ZDkFYUxoGLHHhBA/AeoNruHM0i70/ZI= <<<<<<<<<<
- identity: {}
cloud-user@smf-data-master-1:~$
From Master-3:
cloud-user@smf-data-master-3:~$ cat /data/kubernetes/secrets.conf
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: XK+7mbh3YEnMdqswtySQ1d6QRehg+K6/J1d2e3EnMvI= <<<<<<<<
- identity: {}
cloud-user@smf-data-master-3:~$
恢复步骤
- 在恢复过程中，将master-3的当前密钥复制到备份文件：
cloud-user@smf-data-master-3:~$ sudo cp /data/kubernetes/secrets.conf /data/kubernetes/secrets.conf-bkp
2.编辑密钥，在Master-3中配置密钥，并将secret值更改为与其他主节点中相同的值。
cloud-user@smf-data-master-3:~$ sudo vim /data/kubernetes/secrets.conf
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: XK+7mbh3YEnMdqswtySQ1d6QRehg+K6/J1d2e3EnMvI= <---- Change this value to “BG5hleucjlD5ZDkFYUxoGLHHhBA/AeoNruHM0i70/ZI=“ as in other Master nodes
- identity: {}
3.在Master-3上重新启动kube-apiserver容器：
cloud-user@smf-data-master-3:~$ sudo docker ps -f "name=k8s_kube-apiserver" -q | xargs sudo docker restart
过帐检查
从主设备验证Kubernetes:
cloud-user@pod-name-smf-master-1:~$ kubectl get pods -A -o wide | grep kube-apiserver
现在，所有Pod必须已启动，并且必须在不重新启动的情况下运行。
反馈