简介
本文档介绍在云本机部署平台(CNDP)中部署基于冗余配置管理器(RCM)的一站式(AIO)服务器用于任何硬件问题或维护相关活动的返回材料授权(RMA)的详细过程。
先决条件
要求
Cisco 建议您了解以下主题:
使用的组件
本文档中的信息基于RCM版本 — rcm.2021.02.1.i18
本文档中的信息都是基于特定实验室环境中的设备编写的。本文档中使用的所有设备最初均采用原始(默认)配置。如果您的网络处于活动状态,请确保您了解所有命令的潜在影响。
了解RCM IP方案
本文档介绍RCM设计,包括两个AIO节点和两个RCM操作程序,每个AIO节点一个RCM CEE。
本文中RMA的目标RCM AIO节点是AIO-1(AI0301),它同时包含处于主状态的RCM操作程序。
POD_NAME |
NODE_NAME |
IP_ADDRESS |
DEVICE_TYPE |
OS_TYPE |
UP0300 |
RCE301 |
10.1.2.9 |
RCM_CEE_AIO_1 |
opscenter |
UP0300 |
RCE302 |
10.1.2.10 |
RCM_CEE_AIO_2 |
opscenter |
UP0300 |
AI0301 |
10.1.2.7 |
RCM_K8_AIO_1 |
linux |
UP0300 |
AI0302 |
10.1.2.8 |
RCM_K8_AIO_2 |
linux |
UP0300 |
RM0301 |
10.1.2.3 |
RCM1_ACTIVE |
opscenter |
UP0300 |
RM0302 |
10.1.2.4 |
RCM1_STANDBY |
opscenter |
UP0300 |
RM0303 |
10.1.2.5 |
RCM2_ACTIVE |
opscenter |
UP0300 |
RM0304 |
10.1.2.6 |
RCM2_STANDBY |
opscenter |
备份过程
备份配置
首先,从目标AIO节点上运行的RCM操作程序收集运行配置的配置备份。
# show running-config | nomore
从目标AIO节点上运行的RCM CEE操作程序收集运行配置。
# show running-config | nomore
预检查过程
预检查AIO
收集两个AIO节点的命令输出,并验证所有Pod是否都处于Running状态。
# kubectl get ns
# kubectl get pods -A -o wide
预检查输出示例
请注意,两个RCM操作员和一个在AIO-1节点上运行的RCM CEE操作中心
cloud-user@up0300-aio-1-master-1:~$ kubectl get ns
NAME STATUS AGE
cee-rce301 Active 110d <--
default Active 110d
istio-system Active 110d
kube-node-lease Active 110d
kube-public Active 110d
kube-system Active 110d
nginx-ingress Active 110d
rcm-rm0301 Active 110d <--
rcm-rm0303 Active 110d <--
registry Active 110d
smi-certs Active 110d
smi-node-label Active 110d
smi-vips Active 110d
cloud-user@up0300-aio-1-master-1:~$
登录到AIO-1的两个RCM操作中心并验证状态。
[up0300-aio-1/rm0301] rcm# rcm show-status
message :
{"status":[" Fri Oct 29 07:21:11 UTC 2021 : State is MASTER"]}
[up0300-aio-1/rm0301] rcm#
[up0300-aio-1/rm0303] rcm# rcm show-status
message :
{"status":[" Fri Oct 29 07:22:18 UTC 2021 : State is MASTER"]}
[up0300-aio-1/rm0303] rcm#
在AIO-2节点上重复相同的步骤,其中其他两个RCM操作符对应于AIO-1节点。
cloud-user@up0300-aio-2-master-1:~$ kubectl get ns
NAME STATUS AGE
cee-rce302 Active 105d <--
default Active 105d
istio-system Active 105d
kube-node-lease Active 105d
kube-public Active 105d
kube-system Active 105d
nginx-ingress Active 105d
rcm-rm0302 Active 105d <--
rcm-rm0304 Active 105d <--
registry Active 105d
smi-certs Active 105d
smi-node-label Active 105d
smi-vips Active 105d
cloud-user@up0300-aio-2-master-1:~$
登录到AIO-2的两个RCM操作中心并验证状态。
[up0300-aio-2/rm0302] rcm# rcm show-status
message :
{"status":[" Fri Oct 29 09:32:54 UTC 2021 : State is BACKUP"]}
[up0300-aio-2/rm0302] rcm#
[up0300-aio-2/rm0304] rcm# rcm show-status
message :
{"status":[" Fri Oct 29 09:33:51 UTC 2021 : State is BACKUP"]}
[up0300-aio-2/rm0304] rcm#
执行过程
关闭AIO节点之前在RCM上执行的步骤
- 由于AIO-1上的两个RCM都是主设备,因此您可以将其迁移到BACKUP。
a.为此,您必须先在活动RCM上执行rcm migrate primary命令,然后再关闭AIO-1服务器。
[up0300-aio-1/rm0301] rcm# rcm migrate primary
[up0300-aio-1/rm0303] rcm# rcm migrate primary
b.验证AIO-1上的状态现在是否为BACKUP。
[up0300-aio-1/rm0301] rcm# rcm show-status
[up0300-aio-1/rm0303] rcm# rcm show-status
c.验证AIO-2上的状态现在是否为MASTER,并确保其为MASTER。
[up0300-aio-1/rm0302] rcm# rcm show-status
[up0300-aio-1/rm0304] rcm# rcm show-status
d.对rm0301和rm0303执行RCM关闭。
[up0300-aio-2/rm0301] rcm# config
Entering configuration mode terminal
[up0300-aio-2/rm0301] rcm(config)# system mode shutdown
[up0300-aio-1/rce301] rcm(config)# commit comment <CRNUMBER>
[up0300-aio-2/rm0303] rcm# config
Entering configuration mode terminal
[up0300-aio-2/rm0303] rcm(config)# system mode shutdown
[up0300-aio-1/rce303] rcm(config)# commit comment <CRNUMBER>
2.我们还必须关闭在AIO-1上运行的CEE操作(使用的命令)。
[up0300-aio-1/rce301] cee# config
Entering configuration mode terminal
[up0300-aio-1/rce301] cee(config)# system mode shutdown
[up0300-aio-1/rce301] cee(config)# commit comment <CRNUMBER>
[up0300-aio-1/rce301] cee(config)# exit
等待几分钟,然后检查系统显示0.0%。
[up0300-aio-1/rce301] cee# show system
3.确认除documentation、smart-agent、ops-center-rcm和ops-center-cee pod外,没有RCM和CEE命名空间的pod
# kubectl get pods -n rcm-rm0301 -o wide
# kubectl get pods -n rcm-rm0303 -o wide
# kubectl get pods -n cee-rce302 -o wide
关闭AIO节点之前在Kubernetes节点上执行的步骤
排出Kubernetes节点,以便正常终止关联的Pod和服务。调度程序不再选择此Kubernetes节点并从该节点中移除Pod。请一次清空一个节点。
登录到SMI集群管理器。
cloud-user@bot-deployer-cm-primary:~$ kubectl get svc -n smi-cm
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cluster-files-offline-smi-cluster-deployer ClusterIP 10.102.108.177 <none> 8080/TCP 78d
iso-host-cluster-files-smi-cluster-deployer ClusterIP 10.102.255.174 192.168.0.102 80/TCP 78d
iso-host-ops-center-smi-cluster-deployer ClusterIP 10.102.58.99 192.168.0.100 3001/TCP 78d
netconf-ops-center-smi-cluster-deployer ClusterIP 10.102.108.194 10.244.110.193 3022/TCP,22/TCP 78d
ops-center-smi-cluster-deployer ClusterIP 10.102.156.123 <none> 8008/TCP,2024/TCP,2022/TCP,7681/TCP,3000/TCP,3001/TCP 78d
squid-proxy-node-port NodePort 10.102.73.130 <none> 3128:31677/TCP 78d
cloud-user@bot-deployer-cm-primary:~$ ssh -p 2024 admin@<Cluster IP of ops-center-smi-cluster-deployer>
Welcome to the Cisco SMI Cluster Deployer on bot-deployer-cm-primary
Copyright © 2016-2020, Cisco Systems, Inc.
All rights reserved.
admin connected from 192.168.0.100 using ssh on ops-center-smi-cluster-deployer-686b66d9cd-nfzx8
[bot-deployer-cm-primary] SMI Cluster Deployer#
[bot-deployer-cm-primary] SMI Cluster Deployer# show clusters
LOCK TO
NAME VERSION
----------------------------
cp0100-smf-data -
cp0100-smf-ims -
cp0200-smf-data -
cp0200-smf-ims -
up0300-aio-1 - <--
up0300-aio-2 -
up0300-upf-data -
up0300-upf-ims -
排出主节点:
[bot-deployer-cm-primary] SMI Cluster Deployer# clusters up0300-aio-1 nodes master-1 actions sync drain remove-node true
This would run drain on the node, disrupting pods running on the node. Are you sure? [no,yes] yes
message accepted
将主–1节点标记为维护模式:
[bot-deployer-cm-primary] SMI Cluster Deployer# config
Entering configuration mode terminal
[bot-deployer-cm-primary] SMI Cluster Deployer(config)# clusters up0300-aio-1
[bot-deployer-cm-primary] SMI Cluster Deployer(config-clusters-up0300-aio-1)# nodes master-1
[bot-deployer-cm-primary] SMI Cluster Deployer(config-nodes-master1)# maintenance true
[bot-deployer-cm-primary] SMI Cluster Deployer(config-nodes-master1)# commit
Commit complete.
[bot-deployer-cm-primary] SMI Cluster Deployer(config-nodes-master1)# end
运行集群同步并监控同步操作的日志:
[bot-deployer-cm-primary] SMI Cluster Deployer# clusters up0300-aio-1 nodes master-1 actions sync
This would run sync. Are you sure? [no,yes] yes
message accepted
[bot-deployer-cm-primary] SMI Cluster Deployer# clusters up0300-aio-1 nodes master-1 actions sync logs
群集同步日志的示例输出:
[installer-master] SMI Cluster Deployer# clusters kali-stacked nodes cmts-worker1-1 actions sync logs
Example Cluster Name: kali-stacked
Example WorkerNode: cmts-worker1
logs 2020-10-06 20:01:48.023 DEBUG cluster_sync.kali-stacked.cmts-worker1: Cluster name: kali-stacked
2020-10-06 20:01:48.024 DEBUG cluster_sync.kali-stacked.cmts-worker1: Node name: cmts-worker1
2020-10-06 20:01:48.024 DEBUG cluster_sync.kali-stacked.cmts-worker1: debug: false
2020-10-06 20:01:48.024 DEBUG cluster_sync.kali-stacked.cmts-worker1: remove_node: true
PLAY [Check required variables] ************************************************
TASK [Gathering Facts] *********************************************************
Tuesday 06 October 2020 20:01:48 +0000 (0:00:00.017) 0:00:00.017 *******
ok: [master3]
ok: [master1]
ok: [cmts-worker1]
ok: [cmts-worker3]
ok: [cmts-worker2]
ok: [master2]
TASK [Check node_name] *********************************************************
Tuesday 06 October 2020 20:01:50 +0000 (0:00:02.432) 0:00:02.450 *******
skipping: [master1]
skipping: [master2]
skipping: [master3]
skipping: [cmts-worker1]
skipping: [cmts-worker2]
skipping: [cmts-worker3]
PLAY [Wait for ready and ensure uncordoned] ************************************
TASK [Cordon and drain node] ***************************************************
Tuesday 06 October 2020 20:01:51 +0000 (0:00:00.144) 0:00:02.594 *******
skipping: [master1]
skipping: [master2]
skipping: [master3]
skipping: [cmts-worker2]
skipping: [cmts-worker3]
TASK [upgrade/cordon : Cordon/Drain/Delete node] *******************************
Tuesday 06 October 2020 20:01:51 +0000 (0:00:00.205) 0:00:02.800 *******
changed: [cmts-worker1 -> 172.22.18.107]
PLAY RECAP *********************************************************************
cmts-worker1 : ok=2 changed=1 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
cmts-worker2 : ok=1 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0
cmts-worker3 : ok=1 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0
master1 : ok=1 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0
master2 : ok=1 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0
master3 : ok=1 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0
Tuesday 06 October 2020 20:02:29 +0000 (0:00:38.679) 0:00:41.479 *******
===============================================================================
2020-10-06 20:02:30.057 DEBUG cluster_sync.kali-stacked.cmts-worker1: Cluster sync successful
2020-10-06 20:02:30.058 DEBUG cluster_sync.kali-stacked.cmts-worker1: Ansible sync done
2020-10-06 0:02:30.058 INFO cluster_sync.kali-stacked.cmts-worker1: _sync finished. Opening lock
服务器维护程序
从CIMC正常关闭服务器电源。按照硬件MoP中的定义继续硬件相关的维护活动,并确保在服务器通电后通过所有运行状况检查。
注意:本文不包括服务器的硬件或维护活动MoP,因为它们与问题陈述不同
Kubernetes恢复过程
在AIO节点上加电后对Kubernetes节点执行的步骤
登录到SMI集群管理器:
cloud-user@bot-deployer-cm-primary:~$ kubectl get svc -n smi-cm
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cluster-files-offline-smi-cluster-deployer ClusterIP 10.102.108.177 <none> 8080/TCP 78d
iso-host-cluster-files-smi-cluster-deployer ClusterIP 10.102.255.174 192.168.0.102 80/TCP 78d
iso-host-ops-center-smi-cluster-deployer ClusterIP 10.102.58.99 192.168.0.100 3001/TCP 78d
netconf-ops-center-smi-cluster-deployer ClusterIP 10.102.108.194 10.244.110.193 3022/TCP,22/TCP 78d
ops-center-smi-cluster-deployer ClusterIP 10.102.156.123 <none> 8008/TCP,2024/TCP,2022/TCP,7681/TCP,3000/TCP,3001/TCP 78d
squid-proxy-node-port NodePort 10.102.73.130 <none> 3128:31677/TCP 78d
cloud-user@bot-deployer-cm-primary:~$ ssh -p 2024 admin@<ClusterIP of ops-center-smi-cluster-deployer>
Welcome to the Cisco SMI Cluster Deployer on bot-deployer-cm-primary
Copyright © 2016-2020, Cisco Systems, Inc.
All rights reserved.
admin connected from 192.168.0.100 using ssh on ops-center-smi-cluster-deployer-686b66d9cd-nfzx8
[bot-deployer-cm-primary] SMI Cluster Deployer#
[bot-deployer-cm-primary] SMI Cluster Deployer# show clusters
LOCK TO
NAME VERSION
----------------------------
cp0100-smf-data -
cp0100-smf-ims -
cp0200-smf-data -
cp0200-smf-ims -
up0300-aio-1 - <--
up0300-aio-2 -
up0300-upf-data -
up0300-upf-ims -
关闭要重新添加到集群中的master-1的维护标志。
[bot-deployer-cm-primary] SMI Cluster Deployer# config
Entering configuration mode terminal
[bot-deployer-cm-primary] SMI Cluster Deployer(config)# clusters up0300-aio-1
[bot-deployer-cm-primary] SMI Cluster Deployer(config-clusters-up0300-aio-1)# nodes master-1
[bot-deployer-cm-primary] SMI Cluster Deployer(config-nodes-master-1)# maintenance false
[bot-deployer-cm-primary] SMI Cluster Deployer(config-nodes-master-1)# commit
Commit complete.
[bot-deployer-cm-primary] SMI Cluster Deployer(config-nodes-master-1)# end
使用群集同步操作恢复主节点Pod和服务。
[bot-deployer-cm-primary] SMI Cluster Deployer# clusters up0100-aio-1 nodes master-1 actions sync run debug true
This would run sync. Are you sure? [no,yes] yes
message accepted
监视同步操作的日志。
[bot-deployer-cm-primary] SMI Cluster Deployer# clusters up0100-aio-1 nodes master-1 actions sync logs
检查AIO-1主设备的集群状态。
[bot-deployer-cm-primary] SMI Cluster Deployer# clusters up0300-aio-1 actions k8s cluster-status
示例输出:
[installer-] SMI Cluster Deployer# clusters kali-stacked actions k8s cluster-status
pods-desired-count 67
pods-ready-count 67
pods-desired-are-ready true
etcd-healthy true
all-ok true
RCM恢复过程
在CEE和RCM运营中心执行以恢复应用的步骤
将CEE opscenter和RCM opscenter更新为运行模式。
配置rce301的运行模式。
[up0300-aio-1/rce301] cee# config
Entering configuration mode terminal
[up0300-aio-1/rce301] cee(config)# system mode running
[up0300-aio-1/rce301] cee(config)# commit comment <CRNUMBER>
[up0300-aio-1/rce301] cee(config)# exit
等待几分钟,检查系统是否处于100.0%状态。
[up0300-aio-1/rce301] cee# show system
配置rm0301的运行模式。
[up0300-aio-2/rm0301] rcm# config
Entering configuration mode terminal
[up0300-aio-2/rm0301] rcm(config)# system mode running
[up0300-aio-1/rce301] rcm(config)# commit comment <CRNUMBER>
等待几分钟,验证系统是否为100.0%。
[up0300-aio-1/rm0301] cee# show system
配置rm0303的运行模式。
[up0300-aio-2/rm0303] rcm# config
Entering configuration mode terminal
[up0300-aio-2/rm0303] rcm(config)# system mode running
[up0300-aio-1/rce303] rcm(config)# commit comment <CRNUMBER>
等待几分钟,检查系统是否处于100.0%状态。
[up0300-aio-1/rm0303] cee# show system
验证过程
使用这些命令验证两个AIO节点上的Pod均处于UP和Running状态。
on AIO nodes:
kubectl get ns
kubectl get pods -A -o wide
on RCM ops-centers:
rcm show-status