简介
本文档介绍如何实施针对用户微服务基础设施(SMI)通用执行环境(CEE)设备池(POD)(pgpool)重启问题的解决方法。
先决条件
要求
Cisco 建议您了解以下主题：
- 思科SMI CEE(Ultra Cloud Core CEE)
- 5G云本地部署平台(CNDP)或SMI裸机(BM)架构
- 多克斯和库伯内特斯
使用的组件
本文档中的信息基于以下软件和硬件版本：
- SMI 2020.02.2.35
- 库贝内特斯v1.21.0
本文档中的信息都是基于特定实验室环境中的设备编写的。本文档中使用的所有设备最初均采用原始（默认）配置。如果您的网络处于活动状态，请确保您了解所有命令的潜在影响。
背景信息
什么是SMI?
思科SMI是云技术和标准的分层堆栈，支持来自思科移动、电缆和宽带网关(BNG)业务单元的基于微服务的应用，所有这些业务单元都具有相似的用户管理功能和类似的datastore要求。
属性包括：
- 层云堆栈（技术和标准），可提供自上而下的部署，并适应当前客户云基础设施。
- CEE由所有应用共享，用于非应用功能（数据存储、部署、配置、遥测和报警）。 这为所有客户触点和集成点提供一致的交互和体验。
- 应用和中东欧部署在微服务容器中，并与智能服务网状网连接。
- 用于部署、配置和管理的外露API，以实现自动化。
什么是SMI CEE?
CEE是为监控部署在SMI上的移动和电缆应用而开发的软件解决方案。CEE以集中方式从应用中获取信息（关键指标），供工程师调试和故障排除。
CEE是为所有应用安装的通用工具集。它配备了专用的运营中心，该中心提供用户界面(CLI)和API来管理监控工具。每个集群只有一个CEE可用。
什么是CEE POD?
POD是在Kubernetes群集上运行的进程。POD封装一个称为容器的粒度单元。POD包含一个或多个容器。
Kubernetes在一个节点上部署一个或多个POD，该节点可以是物理或虚拟机。每个POD都有一个具有内部IP地址和端口空间的离散身份。但是，POD中的容器可以共享存储和网络资源。CEE拥有许多功能独特的POD。Pgpool和出口是多个CEE POD之一。
什么是pgpool POD?
Pgpool管理连接、复制、负载平衡等的Postgres资源池。Pgpool是一个中间件，在PostgreSQL服务器和PostgreSQL数据库之间工作。
什么是Postgres POD?
Postgres支持结构化查询语言(SQL)数据库，该数据库具有存储警报和Grafana控制面板的冗余。
问题
pgpool POD定期重新启动，而postgresql POD运行时无问题。
要显示警报，请输入以下命令：
show alerts active summary | include "POD_|k8s-pod-"
此处显示了来自中东欧地区的示例警报。
[pod-name-smf-data/podname] cee# show alerts active summary | include "POD_|k8s-pod-"
k8s-pod-crashing-loop 1d9d2b113073 critical 12-15T21:47:39 pod-name-smf-data-mas
Pod cee-podname/grafana-65cbdb9846-krgfq (grafana) is restarting 1.03 times / 5 minutes.
POD_Restarted 04d42efb81de major 12-15T21:45:44 pgpool-67f48f6565-vjt Container=
k8s_pgpool_pgpool-67f48f6565-vjttd_cee-podname_a9f68607-eac4-40a9-86ef-db8176e0a22a_1474 of pod= pgpool-...
POD_Restarted f7657a0505c2 major 12-15T21:45:44 postgres-0 Container=
k8s_postgres_postgres-0_cee-podname_59e0a768-6870-4550-8db3-32e2ab047ce2_1385 of pod= postgres-0 in name...
POD_Restarted 6e57ae945677 major 12-15T21:45:44 alert-logger-d96644d4 Container=
k8s_alert-logger_alert-logger-d96644d4-dsc8h_cee-podname_2143c464-068a-418e-b5dd-ce1075b9360e_2421 of po...
k8s-pod-crashing-loop 5b8e6a207aad critical 12-15T21:45:09 pod-name-smf-data-mas Pod
cee-podname/pgpool-67f48f6565-vjttd (pgpool) is restarting 1.03 times / 5 minutes.
POD_Down 45a6b9bf73dc major 12-15T20:30:44 pgpool-67f48f6565-qbw Pod= pgpool-67f48f6565-qbw52 in namespace=
cee-podname is DOWN for more than 15min
POD_Down 4857f398a0ca major 12-15T16:40:44 pgpool-67f48f6565-vjt Pod= pgpool-67f48f6565-vjttd in namespace=
cee-podname is DOWN for more than 15min
k8s-pod-not-ready fc65254c2639 critical 12-11T21:07:29 pgpool-67f48f6565-qbw Pod
cee-podname/pgpool-67f48f6565-qbw52 has been in a non-ready state for longer than 1 minute.
k8s-pod-not-ready 008b859e7333 critical 12-11T16:35:49 pgpool-67f48f6565-vjt Pod
cee-podname/pgpool-67f48f6565-vjttd has been in a non-ready state for longer than 1 minute.
故障排除
从Kubernetes主设备输入以下命令：
kubectl describe pods -n
postgres-0"
POD说明的输出示例如下所示。输出被截断。
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 14m default-scheduler Successfully assigned cee-pod-name-l1/postgres-2
to pod-name-master-3
Normal Pulling 14m kubelet Pulling image "docker.10.192.x.x.nip.io/cee-2020.02.2.i38/
smi-libraries/postgresql/2020.02.2/postgres:1.3.0-946d87d"
Normal Pulled 13m kubelet Successfully pulled image "docker.10.192.x.x.nip.io/cee-2020.02.2.i38/
smi-libraries/postgresql/2020.02.2/postgres:1.3.0-946d87d" in 29.048094722s
Warning Unhealthy 12m kubelet Readiness probe failed: [bin][h][ir] >>> [2021-10-11 18:09:48]
pod is not ready
Warning Unhealthy 10m kubelet Readiness probe failed: [bin][h][ir] >>> [2021-10-11 18:11:18]
pod is not ready
Warning Unhealthy 10m kubelet Readiness probe failed: [bin][h][ir] >>> [2021-10-11 18:11:48]
pod is not ready
Warning Unhealthy 9m49s kubelet Readiness probe failed: [bin][h][ir] >>> [2021-10-11 18:12:18]
pod is not ready
Warning Unhealthy 9m19s kubelet Readiness probe failed: [bin][h][ir] >>> [2021-10-11 18:12:48]
pod is not ready
Warning Unhealthy 8m49s kubelet Readiness probe failed: [bin][h][ir] >>> [2021-10-11 18:13:18]
pod is not ready
Warning Unhealthy 8m19s kubelet Readiness probe failed: [bin][h][ir] >>> [2021-10-11 18:13:48]
pod is not ready
Warning Unhealthy 7m49s kubelet Readiness probe failed: [bin][h][ir] >>> [2021-10-11 18:14:18]
pod is not ready
Warning Unhealthy 7m19s kubelet Readiness probe failed: [bin][h][ir] >>> [2021-10-11 18:14:48]
pod is not ready
Warning BackOff 6m44s kubelet Back-off restarting failed container
或者
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 13m default-scheduler 0/5 nodes are available: 2 node(s)
didn't match Pod's node affinity/selector, 3 node(s) didn't find available persistent
volumes to bind.
Normal Scheduled 13m default-scheduler Successfully assigned cee-pod-name-l1/postgres-0
to pod-name-master-1
Warning FailedScheduling 13m default-scheduler 0/5 nodes are available: 2 node(s)
didn't match Pod's node affinity/selector, 3 node(s) didn't find available
persistent volumes to bind.
Normal Pulling 13m kubelet Pulling image "docker.10.192.x.x.nip.io/cee-2020.02.2.i38/
smi-libraries/postgresql/2020.02.2/postgres:1.3.0-946d87d"
Normal Pulled 12m kubelet Successfully pulled image "docker.10.192.x.x.nip.io/
cee-2020.02.2.i38/smi-libraries/postgresql/2020.02.2/postgres:1.3.0-946d87d"
in 43.011763302s
Warning Unhealthy 7m20s kubelet Liveness probe failed: [bin][h][imm] >>>
[2021-10-11 18:09:16] My name is pg-postgres-0
解决方法
注意：此过程不会导致应用程序出现任何停机。
关闭CEE
要关闭CEE，请从CEE输入以下命令：
[pod-name-smf-data/podname] cee#
[pod-name-smf-data/podname] cee# config terminal
Entering configuration mode terminal
[pod-name-smf-data/podname] cee(config)# system mode shutdown
[pod-name-smf-data/podname] cee(config)# commit
Commit complete.
等待系统达到100%
从文件夹中删除内容
从master-vip，SSH到每个主VM，并删除以下文件夹的内容：/data/cee-podname/data-postgres-[0-2]。
Master 1
cloud-user@pod-name-smf-data-master-1:~$ sudo rm -rf /data/cee-podname/data-postgres-0
Master 2
cloud-user@pod-name-smf-data-master-2:~$ sudo rm -rf /data/cee-podname/data-postgres-1
Master 3
cloud-user@pod-name-smf-data-master-3:~$ sudo rm -rf /data/cee-podname/data-postgres-2
恢复CEE
要恢复CEE，请从CEE输入以下命令：
[pod-name-smf-data/podname] cee#
[pod-name-smf-data/podname] cee# config terminal
Entering configuration mode terminal
[pod-name-smf-data/podname] cee(config)# system mode running
[pod-name-smf-data/podname] cee(config)# commit
Commit complete.
等待系统达到100%。
过帐检查
从主设备验证Kubernetes。
cloud-user@pod-name-smf-data-master-1:~$ kubectl get pods -A -o wide | egrep 'postgres|pgpool'
All pods should display up and running without any restarts
验证是否已从CEE清除警报
要验证警报是否已从CEE清除，请输入以下命令：
show alerts active summary | include "POD_|k8s-pod-"
此外，您可以输入此命令，以确保有一个主数据库和两个备用数据库：
echo "0----------------------------------";kubectl
exec -it postgres-0 -n $(kubectl get pods -A | grep postgres | awk '{print $1}' | head -1)
-- /usr/local/bin/cluster/healthcheck/is_major_master.sh;echo "1--------------------------
--------";kubectl exec -it postgres-1 -n $(kubectl get pods -A | grep postgres | awk '{print $1}'
| head -1) -- /usr/local/bin/cluster/healthcheck/is_major_master.sh;echo "2---------------
-------------------"; kubectl exec -it postgres-2 -n $(kubectl get pods -A | grep postgres |
awk '{print $1}' | head -1) -- /usr/local/bin/cluster/healthcheck/is_major_master.sh;
预期输出示例为：
cloud-user@pod-name-smf-data-master-1:~$ echo "0----------------------------------";kubectl
exec -it postgres-0 -n $(kubectl get pods -A | grep postgres | awk '{print $1}' | head -1)
-- /usr/local/bin/cluster/healthcheck/is_major_master.sh;echo "1--------------------------
--------";kubectl exec -it postgres-1 -n $(kubectl get pods -A | grep postgres | awk '{print $1}'
| head -1) -- /usr/local/bin/cluster/healthcheck/is_major_master.sh;echo "2---------------
-------------------"; kubectl exec -it postgres-2 -n $(kubectl get pods -A | grep postgres |
awk '{print $1}' | head -1) -- /usr/local/bin/cluster/healthcheck/is_major_master.sh;
0----------------------------------
[bin][h][imm] >>> [2021-12-15 22:05:18] My name is pg-postgres-0
[bin][h][imm] >>> My state is good.
[bin][h][imm] >>> I'm not a master, nothing else to do!
1----------------------------------
[bin][h][imm] >>> [2021-12-15 22:05:19] My name is pg-postgres-1
[bin][h][imm] >>> My state is good.
[bin][h][imm] >>> I think I'm master. Will ask my neighbors if they agree.
[bin][h][imm] >>> Will ask nodes from PARTNER_NODES list
[bin][h][imm] >>> Checking node pg-postgres-0
[bin][h][imm] >>>>>>>>> Count of references to potential master pg-postgres-1 is 1 now
[bin][h][imm] >>> Checking node pg-postgres-1
[bin][h][imm] >>> Checking node pg-postgres-2
[bin][h][imm] >>>>>>>>> Count of references to potential master pg-postgres-1 is 2 now
[bin][h][imm] >>> Potential masters got references:
[bin][h][imm] >>>>>> Node: pg-postgres-1, references: 2
[bin][h][imm] >>> I have 2/2 incoming reference[s]!
[bin][h][imm] >>>> 2 - Does anyone have more?
[bin][h][imm] >>> Yahoo! I'm real master...so I think!
2----------------------------------
[bin][h][imm] >>> [2021-12-15 22:05:21] My name is pg-postgres-2
[bin][h][imm] >>> My state is good.
[bin][h][imm] >>> I'm not a master, nothing else to do!
反馈