更换OSPD服务器UCS 240M4 - CPAR

简介

本文档介绍在Ultra-M设置中更换托管OpenStack平台导向器(OSPD)的故障服务器所需的步骤。 此过程适用于使用NEWTON版本的OpenStack环境，其中ESC不管理Cisco Prime Access Registrar(CPAR),CPAR直接安装在OpenStack上部署的VM上。

背景信息

Ultra-M是预打包和验证的虚拟化移动数据包核心解决方案，旨在简化VNF的部署。OpenStack是Ultra-M的虚拟化基础设施管理器(VIM)，由以下节点类型组成：

• 计算
• 对象存储磁盘 — 计算（OSD — 计算）
• 控制器
• OSPD

此图中描述了Ultra-M的高级体系结构和涉及的组件：

本文档面向熟悉Cisco Ultra-M平台的思科人员，并详细介绍在OpenStack和Redhat OS中执行所需的步骤。 

---

注意：为了定义本文档中的步骤，我们考虑了Ultra M 5.1.x版本。

---

缩写

MOP	方法
OSD	对象存储磁盘
OSPD	OpenStack平台导向器
硬盘	硬盘驱动器
SSD	固态驱动器
VIM	虚拟基础设施管理器
虚拟机	虚拟机
EM	元素管理器
UAS	超自动化服务
UUID	通用唯一IDentifier

MoP的工作流

替换过程的高级工作流程

先决条件

状态检查

在更换OSPD服务器之前，必须检查Red Hat OpenStack平台环境的当前状态并确保其正常运行，以避免在更换过程开启时出现问题。 

1.检查OpenStack堆栈的状态和节点列表：

[stack@director ~]$ source stackrc 
[stack@director ~]$ openstack stack list --nested
[stack@director ~]$ ironic node-list
[stack@director ~]$ nova list

2.检查OSP-D节点中是否所有下云服务都处于已加载、活动和运行状态：

[stack@al03-pod2-ospd ~]$ systemctl list-units "openstack*" "neutron*" "openvswitch*"
 UNIT LOAD ACTIVE SUB DESCRIPTION
 neutron-dhcp-agent.service loaded active running OpenStack Neutron DHCP Agent
 neutron-metadata-agent.service loaded active running OpenStack Neutron Metadata Agent
 neutron-openvswitch-agent.service loaded active running OpenStack Neutron Open vSwitch Agent
 neutron-server.service loaded active running OpenStack Neutron Server
 openstack-aodh-evaluator.service loaded active running OpenStack Alarm evaluator service
 openstack-aodh-listener.service loaded active running OpenStack Alarm listener service
 openstack-aodh-notifier.service loaded active running OpenStack Alarm notifier service
 openstack-ceilometer-central.service loaded active running OpenStack ceilometer central agent
 openstack-ceilometer-collector.service loaded active running OpenStack ceilometer collection service
 openstack-ceilometer-notification.service loaded active running OpenStack ceilometer notification agent
 openstack-glance-api.service loaded active running OpenStack Image Service (code-named Glance) API server
 openstack-glance-registry.service loaded active running OpenStack Image Service (code-named Glance) Registry server
 openstack-heat-api-cfn.service loaded active running Openstack Heat CFN-compatible API Service
 openstack-heat-api.service loaded active running OpenStack Heat API Service
 openstack-heat-engine.service loaded active running Openstack Heat Engine Service
 openstack-ironic-api.service loaded active running OpenStack Ironic API service
 openstack-ironic-conductor.service loaded active running OpenStack Ironic Conductor service
 openstack-ironic-inspector-dnsmasq.service loaded active running PXE boot dnsmasq service for Ironic Inspector
 openstack-ironic-inspector.service loaded active running Hardware introspection service for OpenStack Ironic
 openstack-mistral-api.service loaded active running Mistral API Server
 openstack-mistral-engine.service loaded active running Mistral Engine Server
 openstack-mistral-executor.service loaded active running Mistral Executor Server
 openstack-nova-api.service loaded active running OpenStack Nova API Server
 openstack-nova-cert.service loaded active running OpenStack Nova Cert Server
 openstack-nova-compute.service loaded active running OpenStack Nova Compute Server
 openstack-nova-conductor.service loaded active running OpenStack Nova Conductor Server
 openstack-nova-scheduler.service loaded active running OpenStack Nova Scheduler Server
 openstack-swift-account-reaper.service loaded active running OpenStack Object Storage (swift) - Account Reaper
 openstack-swift-account.service loaded active running OpenStack Object Storage (swift) - Account Server
 openstack-swift-container-updater.service loaded active running OpenStack Object Storage (swift) - Container Updater
 openstack-swift-container.service loaded active running OpenStack Object Storage (swift) - Container Server
 openstack-swift-object-updater.service loaded active running OpenStack Object Storage (swift) - Object Updater
 openstack-swift-object.service loaded active running OpenStack Object Storage (swift) - Object Server
 openstack-swift-proxy.service loaded active running OpenStack Object Storage (swift) - Proxy Server
 openstack-zaqar.service loaded active running OpenStack Message Queuing Service (code-named Zaqar) Server
 openstack-zaqar@1.service loaded active running OpenStack Message Queuing Service (code-named Zaqar) Server Instance 1
 openvswitch.service loaded active exited Open vSwitch

LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
lines 1-43 
lines 2-44 37 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
lines 4-46/46 (END) lines 4-46/46 (END) lines 4-46/46 (END) lines 4-46/46 (END) lines 4-46/46 (END)

备份

1.在执行备份过程之前，请确认您有足够的可用磁盘空间。预计该目标至少为3.5 GB。

[stack@director ~]$ df -h

2.以根用户身份运行这些命令，以便将数据从下云节点备份到名为undercloud-backup-[timestamp].tar.gz的文件。

[root@director ~]# mysqldump --opt --all-databases > /root/undercloud-all-databases.sql
[root@director ~]# tar --xattrs -czf undercloud-backup-`date +%F`.tar.gz /root/undercloud-all-databases.sql 
/etc/my.cnf.d/server.cnf /var/lib/glance/images /srv/node /home/stack
tar: Removing leading `/' from member names

安装新OSPD节点

UCS服务器安装

1.安装新UCS C240 M4服务器的步骤和初始设置步骤可参阅《Cisco UCS C240 M4服务器安装和服务指南》。

2.在安装服务器后，将硬盘插入各自的插槽作为旧服务器。

3.使用思科集成管理控制器(CIMC)IP登录到服务器。

4.如果固件与之前使用的推荐版本不同，请执行BIOS升级。BIOS升级步骤如下：Cisco UCS C系列机架式服务器BIOS升级指南。

5.检验物理驱动器的状态。它必须是未配置的正常。导航至存储> Cisco 12G SAS模块化RAID控制器(SLOT-HBA)>物理驱动器信息，如图所示。

6.从RAID级别为1的物理驱动器创建虚拟驱动器。导航至Storage > Cisco 12G SAS模块化RAID控制器(SLOT-HBA)> Controller Info > Create Virtual Drive from Unused Physical Drives（从未使用的物理驱动器创建虚拟驱动器），如图所示。

7.选择VD并配置“设置为引导驱动器”，如图所示。

8.启用IPMI over LAN。导航至Admin > Communication Services > Communication Services，如图所示。

  

9.禁用超线程。如图所示，导航至计算> BIOS >配置BIOS >高级>处理器配置。

---

注意：此处显示的图像和本节中提及的配置步骤均参考固件版本3.0(3e)，如果您使用其他版本，可能会略有变化。

---

Redhat安装

安装Red Hat ISO映像

1.登录OSP-D服务器。

2.启动KVM控制台。

3.导航至 虚拟媒体>激活虚拟设备。接受会话并启用“记住将来连接的设置”。

4.导航至 虚拟媒体> 映射CD/DVD并映射Red Hat ISO映像。

5.导航至 电源> 按顺序重置系统(热启动) 重新启动系统。

6.重新启动后，按 F6选择思科vKVM映射vDVD1.22并按Enter。

安装RHEL

根据备份准备云下安装

当计算机安装了RHEL 7.3并且处于干净状态后，请重新启用安装和运行指挥交换机所需的所有订用/存储库。

1.主机名配置。

[root@director ~]$sudo hostnamectl set-hostname <FQDN_hostname> 
[root@director ~]$sudo hostnamectl set-hostname --transient <FQDN_hostname>

2.编辑/etc/hosts文件。

[root@director ~]$ vi /etc/hosts
<ospd_external_address>  <server_hostname>  <FQDN_hostname>
192.168.247.142  pod2-stack-ospd pod2-stack-ospd.cisco.com

3.验证主机名。

[root@director ~]$ cat /etc/hostname
pod2-stack-ospd.cisco.com

4.验证DNS配置。

[root@director ~]$ cat /etc/resolv.conf

#Generated by NetworkManager
nameserver <DNS_IP>

5.修改调配NIC接口。

[root@director ~]$ cat /etc/sysconfig/network-scripts/ifcfg-eno1

DEVICE=eno1
ONBOOT=yes
HOTPLUG=no
NM_CONTROLLED=no
PEERDNS=no
DEVICETYPE=ovs
TYPE=OVSPort
OVS_BRIDGE=br-ctlplane
BOOTPROTO=none
MTU=1500

完成Redhat注册

  

1.下载此软件包，以便将订用管理器配置为使用rh-satellite。

[root@director ~]$ rpm -Uvh http://
     
     
       /pub/katello-ca-consumer-latest.noarch.rpm 
     
[root@director ~]$ subscription-manager config

2.使用RHEL 7.3的此活动键向rh-satellite注册。 

[root@director ~]$subscription-manager register --org="<ORG>" --activationkey="<KEY>"

3.要查看订用。

[root@director ~]$ subscription-manager list –consumed

4.启用与旧OSPD回购相同的存储库。

[root@director ~]$ sudo subscription-manager repos --disable=*
[root@director ~]$ subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-extras-rpms --enable=rh
el-7-server-openstack-10-rpms --enable=rhel-7-server-rh-common-rpms --enable=rhel-ha-for-rhel-7-server-rpm

5.在系统上执行更新以确保您拥有最新的基本系统软件包并重新启动系统。

[root@director ~]$sudo yum update -y
[root@director ~]$sudo reboot

云下恢复

启用订用后，将备份的云下tar文件“undercloud-backup-'date +%F'.tar.gz”导入新的OSP-D服务器根目录/root

1.安装mariadb服务器。

[root@director ~]$ yum install -y mariadb-server

2.提取MariaDB配置文件和数据库备份。以根用户身份执行此操作。

[root@director ~]$ tar -xzC / -f undercloud-backup-$DATE.tar.gz etc/my.cnf.d/server.cnf
[root@director ~]$ tar -xzC / -f undercloud-backup-$DATE.tar.gz root/undercloud-all-databases.sql

3.编辑/etc/my.cnf.d/server.cnf并注释掉bind-address条目（如果存在）。

[root@tb3-ospd ~]# vi /etc/my.cnf.d/server.cnf

4.启动MariaDB服务并临时更新max_allowed_packet设置。

[root@director ~]$ systemctl start mariadb
[root@director ~]$ mysql -uroot -e"set global max_allowed_packet = 16777216;"

5.清除某些权限（稍后重新创建）。

[root@director ~]$ for i in ceilometer glance heat ironic keystone neutron nova;do mysql -e "drop user $i";done
[root@director ~]$ mysql -e 'flush privileges'

---

注意：如果之前在设置中禁用了云顶计服务，请执行以上命令，删除“云顶计”。

---

6.创建堆叠用户帐户。

[root@director ~]$ sudo useradd stack
[root@director ~]$ sudo passwd stack << specify a password
[root@director ~]$ echo "stack ALL=(root) NOPASSWD:ALL" | sudo tee -a /etc/sudoers.d/stack
[root@director ~]$ sudo chmod 0440 /etc/sudoers.d/stack 

7.恢复堆栈用户主目录。

[root@director ~]$ tar -xzC / -f undercloud-backup-$DATE.tar.gz home/stack

8.安装快速概览的基本软件包，然后恢复其数据。

[root@director ~]$ yum install -y openstack-glance openstack-swift
[root@director ~]$ tar --xattrs -xzC / -f undercloud-backup-$DATE.tar.gz srv/node var/lib/glance/images

9.确认数据归正确用户所有。

[root@director ~]$ chown -R swift: /srv/node
[root@director ~]$ chown -R glance: /var/lib/glance/images

10.恢复云下SSL证书（可选 — 仅在设置使用SSL证书时完成）。

[root@director ~]$ tar -xzC / -f undercloud-backup-$DATE.tar.gz etc/pki/instack-certs/undercloud.pem
[root@director ~]$ tar -xzC / -f undercloud-backup-$DATE.tar.gz etc/pki/ca-trust/source/anchors/ca.crt.pem

11.以stackuser身份重新运行云下安装，确保在堆栈用户主目录中运行该安装：

[root@director ~]$ su - stack
[stack@director ~]$ sudo yum install -y python-tripleoclient

12.确认主机名在/etc/hosts中设置正确。

13.重新安装下云。

[stack@director ~]$ openstack undercloud install

<snip>
#############################################################################
Undercloud install complete.

The file containing this installation's passwords is at
/home/stack/undercloud-passwords.conf.

There is also a stackrc file at /home/stack/stackrc.

These files are needed to interact with the OpenStack services, and must be
secured.

#############################################################################

将已恢复的下云重新连接到超云

完成上述步骤后，应期望下云自动恢复其与超云的连接。节点将使用每隔几秒发出的简单HTTP请求，继续轮询业务流程（热）以查找待处理任务。

验证已完成的恢复

使用这些命令对新恢复的环境执行运行状况检查。

[root@director ~]$ su - stack
Last Log in: Tue Nov 28 21:27:50 EST 2017 from 192.182.255.20 on pts/0

[stack@director ~]$ source stackrc
[stack@director ~]$ nova list
+--------------------------------------+--------------------------+--------+------------+-------------+------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------------------------+--------+------------+-------------+------------------------+
| 03f15071-21aa-4bcf-8fdd-acdbde305168 | pod2-stack-compute-0 | ACTIVE | - | Running | ctlplane=192.200.0.106 |
| 1f725ce3-948d-49e9-aed9-b99e73d82644 | pod2-stack-compute-1 | ACTIVE | - | Running | ctlplane=192.200.0.107 |
| fbc13c78-dc06-4ac9-a3c5-595ccc147adc | pod2-stack-compute-2 | ACTIVE | - | Running | ctlplane=192.200.0.119 |
| 3b94e0b1-47dc-4960-b3eb-d02ffe9ae693 | pod2-stack-compute-3 | ACTIVE | - | Running | ctlplane=192.200.0.112 |
| 5dbac94d-19b9-493e-a366-1e2e2e5e34c5 | pod2-stack-compute-4 | ACTIVE | - | Running | ctlplane=192.200.0.116 |
| b896c73f-d2c8-439c-bc02-7b0a2526dd70 | pod2-stack-controller-0 | ACTIVE | - | Running | ctlplane=192.200.0.113 |
| 2519ce67-d836-4e5f-a672-1a915df75c7c | pod2-stack-controller-1 | ACTIVE | - | Running | ctlplane=192.200.0.105 |
| e19b9625-5635-4a52-a369-44310f3e6a21 | pod2-stack-controller-2 | ACTIVE | - | Running | ctlplane=192.200.0.120 |
| 6810c884-1cb9-4321-9a07-192443920f1f | pod2-stack-osd-compute-0 | ACTIVE | - | Running | ctlplane=192.200.0.109 |
| 26d3f7b1-ba97-431f-aa6e-ba91661db45d | pod2-stack-osd-compute-1 | ACTIVE | - | Running | ctlplane=192.200.0.117 |
| 6e4a8aa9-4870-465a-a7e2-0932ff55e34b | pod2-stack-osd-compute-2 | ACTIVE | - | Running | ctlplane=192.200.0.103 |
+--------------------------------------+--------------------------+--------+------------+-------------+------------------------+
<snip>

[stack@director ~]$ ssh heat-admin@192.200.0.113
Last login: Fri Jul 6 09:02:41 2018 from 192.200.0.1 
[heat-admin@pod2-stack-controller-0 ~]$ sudo pcs status
Cluster name: tripleo_cluster
Stack: corosync
Current DC: pod2-stack-controller-2 (version 1.1.15-11.el7_3.4-e174ec8) - partition with quorum
Last updated: Tue Jul 10 10:04:15 2018Last change: Fri Jul 6 09:03:35 2018 by root via crm_attribute on pod2-stack-controller-0

3 nodes and 19 resources configured

Online: [ pod2-stack-controller-0 pod2-stack-controller-1 pod2-stack-controller-2 ]

Full list of resources:

 ip-11.120.0.49(ocf::heartbeat:IPaddr2):Started pod2-stack-controller-1
 Clone Set: haproxy-clone [haproxy]
 Started: [ pod2-stack-controller-0 pod2-stack-controller-1 pod2-stack-controller-2 ]
 Master/Slave Set: galera-master [galera]
 Masters: [ pod2-stack-controller-0 pod2-stack-controller-1 pod2-stack-controller-2 ]
 ip-192.200.0.110(ocf::heartbeat:IPaddr2):Started pod2-stack-controller-1
 ip-11.120.0.44(ocf::heartbeat:IPaddr2):Started pod2-stack-controller-2
 ip-11.118.0.49(ocf::heartbeat:IPaddr2):Started pod2-stack-controller-2
 Clone Set: rabbitmq-clone [rabbitmq]
 Started: [ pod2-stack-controller-0 pod2-stack-controller-1 pod2-stack-controller-2 ]
 ip-10.225.247.214(ocf::heartbeat:IPaddr2):Started pod2-stack-controller-1
 Master/Slave Set: redis-master [redis]
 Masters: [ pod2-stack-controller-2 ]
 Slaves: [ pod2-stack-controller-0 pod2-stack-controller-1 ]
 ip-11.119.0.49(ocf::heartbeat:IPaddr2):Started pod2-stack-controller-2
 openstack-cinder-volume(systemd:openstack-cinder-volume):Started pod2-stack-controller-1

Daemon Status:
 corosync: active/enabled
 pacemaker: active/enabled
 pcsd: active/enabled

[heat-admin@pod2-stack-controller-0 ~]$ sudo ceph status
 cluster eb2bb192-b1c9-11e6-9205-525400330666
 health HEALTH_OK
 monmap e1: 3 mons at {pod2-stack-controller-0=11.118.0.10:6789/0,pod2-stack-controller-1=11.118.0.11:6789/0,pod2-stack-controller-2=11.118.0.12:6789/0}
 election epoch 10, quorum 0,1,2 pod2-stack-controller-0,pod2-stack-controller-1,pod2-stack-controller-2
 osdmap e81: 12 osds: 12 up, 12 in
 flags sortbitwise,require_jewel_osds
 pgmap v23094282: 704 pgs, 6 pools, 809 GB data, 424 kobjects
 2418 GB used, 10974 GB / 13393 GB avail
 704 active+clean
 client io 1841 kB/s wr, 0 op/s rd, 220 op/s wr
[heat-admin@pod2-stack-controller-0 ~]$ exit
logout
Connection to 192.200.0.113 closed.

检查身份服务(Keystone)操作

此步骤通过查询用户列表来验证身份服务操作。

[stack@director ~]$ source stackerc
[stack@director~]$ openstack user list
+----------------------------------+------------------+
| ID | Name |
+----------------------------------+------------------+
| 4d93cc08ed314b87b4c6dce95c19558f | admin |
| c499bd1a65d94029804350e3bca19888 | aodh |
| d9b59136b2794ec1b9c18c4f7b95c674 | ceilometer |
| 4650cdca79b44f2b9d2426b7a9df9495 | glance |
| 4a6724a623f4463c971658462b505c8a | heat |
| e18ae9a5b7f14aefb9e93931dcec476e | ironic |
| 9831160669c44c10b04d6f78b7517a55 | ironic-inspector |
| 6a5d6f02b9a74dfb9f679eecfebb28e0 | mistral |
| 462c8ec91fb0485fbc7dfec6d4ecf480 | neutron |
| f3a66d84e39c47a0bdcd14d24bd72816 | nova |
| 1f16ea9035ab424a88d5f34681d3f893 | swift |
| ab4a1e5f62be4609b290c32dc93bfbce | zaqar |
| fac68a166f5f442284f7d06355369c53 | zaqar-websocket |
+----------------------------------+------------------+

[stack@director ~]$ source <overcloudrc>
[stack@director ~]$ openstack user list
+----------------------------------+------------+
| ID | Name |
+----------------------------------+------------+
| 7285a65e0c8c4447b10875d0e9c9da69 | admin |
| c7c78ac31ec24926b3eed79e7f5bd59e | neutron |
| 191c2a293beb48edb4d2c1160c9a0523 | heat |
| 1445a6ef2ae94b6f9365ffbc6b56a231 | gnocchi |
| fb8b006129554c6b8465310fcfc0ba8c | aodh |
| 5afcb816f93b4f20998c9edaf3b143a0 | nova |
| 37444af1295344809eca6058a782f871 | glance |
| e74d2d9e59f34ddfb4408a8beb88db7b | ceilometer |
| 9baab90178c141478ce7c0084e930f40 | cinder |
| 5a964c651c11402284eecb0e8f40ee33 | heat-cfn |
| 6d49ebae55da432580943d6080efcf90 | swift |
| 41a666fe64c64df593d73b4f467885be | core |
| ab004ba966bf4ddc92783fbb6045e48b | cpar |
| fce5b5c5dc7647ccbc2170b56a64db33 | cpardemo |
| 28ead2dab0ef486ba449b441715a3e04 | pcrf |
| 0aac4107e3d443dc93ca62c3928a1998 | pcrfdemo |
+----------------------------------+------------+

上传映像以供将来节点内省

1.验证/httpboot所有这些文件“inspector.ipxe、agent.kernel、agent.ramdisk”（如果不继续以下步骤以在映像中更新）。

[stack@director ~]$ ls /httpboot
inspector.ipxe
[stack@director ~]$ source stackrc 
[stack@director ~]$ cd images/
[stack@director images]$ openstack overcloud image upload --image-path /home/stack/images
Image "overcloud-full-vmlinuz" is up-to-date, skipping.
Image "overcloud-full-initrd" is up-to-date, skipping.
Image "overcloud-full" is up-to-date, skipping.
Image "bm-deploy-kernel" is up-to-date, skipping.
Image "bm-deploy-ramdisk" is up-to-date, skipping.
[stack@director images]$ ls /httpboot
agent.kernel agent.ramdisk inspector.ipxe
[stack@director images]$ 

重新启动围栏

在OSPD恢复后，围栏将处于停止状态，以下程序将启用围栏。 

[heat-admin@pod2-stack-controller-0 ~]$ sudo pcs property set stonith-enabled=true
[heat-admin@pod2-stack-controller-0 ~]$ sudo pcs status
[heat-admin@pod2-stack-controller-0 ~]$sudo pcs stonith show 

相关信息

• 备份和恢复Director Undercloud

• 隔离控制器节点

• 技术支持和文档 - Cisco Systems