更换OSPD服务器UCS 240M4 - CPS

简介

本文档介绍在Ultra-M设置中更换托管OpenStack平台导向器(OSPD)的故障服务器所需的步骤。

背景信息

Ultra-M是预打包和验证的虚拟化移动数据包核心解决方案，旨在简化VNF的部署。OpenStack是Ultra-M的虚拟化基础设施管理器(VIM)，由以下节点类型组成：

• 计算
• 对象存储磁盘 — 计算（OSD — 计算）
• 控制器
• OpenStack平台 — 导向器(OSPD)

Ultra-M的高级体系结构和涉及的组件如下图所示。

UltraM架构  

注意：Ultra M 5.1.x版本用于定义本文档中的步骤。本文档面向熟悉Cisco Ultra-M平台的思科人员，并详细介绍在更换OSPD服务器时在OpenStack级别执行所需的步骤。

缩写

VNF	虚拟网络功能
ESC	弹性服务控制器
MOP	程序方法
OSD	对象存储磁盘
硬盘	硬盘驱动器
SSD	固态驱动器
VIM	虚拟基础设施管理器
虚拟机	虚拟机
EM	元素管理器
UAS	超自动化服务
UUID	通用唯一IDentifier

MoP的工作流

替换过程的高级工作流程   

先决条件

状态检查

在更换OSPD服务器之前，必须检查Red Hat OpenStack平台环境的当前状态并确保其正常运行，以避免在更换过程开启时出现问题。 

1.检查OpenStack堆栈和节点列表的状态。

[stack@director ~]$ source stackrc 
[stack@director ~]$ openstack stack list --nested
[stack@director ~]$ ironic node-list
[stack@director ~]$ nova list

2.检查OSPD节点中是否所有下云服务都处于加载、活动和运行状态。 

[stack@director ~]$ systemctl list-units "openstack*" "neutron*" "openvswitch*"

UNIT                                       LOAD   ACTIVE SUB     DESCRIPTION

neutron-dhcp-agent.service                 loaded active running OpenStack Neutron DHCP Agent
neutron-openvswitch-agent.service          loaded active running OpenStack Neutron Open vSwitch Agent
neutron-ovs-cleanup.service                loaded active exited  OpenStack Neutron Open vSwitch Cleanup Utility
neutron-server.service                     loaded active running OpenStack Neutron Server
openstack-aodh-evaluator.service           loaded active running OpenStack Alarm evaluator service
openstack-aodh-listener.service            loaded active running OpenStack Alarm listener service
openstack-aodh-notifier.service            loaded active running OpenStack Alarm notifier service
openstack-ceilometer-central.service       loaded active running OpenStack ceilometer central agent
openstack-ceilometer-collector.service     loaded active running OpenStack ceilometer collection service
openstack-ceilometer-notification.service  loaded active running OpenStack ceilometer notification agent
openstack-glance-api.service               loaded active running OpenStack Image Service (code-named Glance) API server
openstack-glance-registry.service          loaded active running OpenStack Image Service (code-named Glance) Registry server
openstack-heat-api-cfn.service             loaded active running Openstack Heat CFN-compatible API Service
openstack-heat-api.service                 loaded active running OpenStack Heat API Service
openstack-heat-engine.service              loaded active running Openstack Heat Engine Service
openstack-ironic-api.service               loaded active running OpenStack Ironic API service
openstack-ironic-conductor.service         loaded active running OpenStack Ironic Conductor service
openstack-ironic-inspector-dnsmasq.service loaded active running PXE boot dnsmasq service for Ironic Inspector
openstack-ironic-inspector.service         loaded active running Hardware introspection service for OpenStack Ironic
openstack-mistral-api.service              loaded active running Mistral API Server
openstack-mistral-engine.service           loaded active running Mistral Engine Server
openstack-mistral-executor.service         loaded active running Mistral Executor Server
openstack-nova-api.service                 loaded active running OpenStack Nova API Server
openstack-nova-cert.service                loaded active running OpenStack Nova Cert Server
openstack-nova-compute.service             loaded active running OpenStack Nova Compute Server
openstack-nova-conductor.service           loaded active running OpenStack Nova Conductor Server
openstack-nova-scheduler.service           loaded active running OpenStack Nova Scheduler Server
openstack-swift-account-reaper.service     loaded active running OpenStack Object Storage (swift) - Account Reaper
openstack-swift-account.service            loaded active running OpenStack Object Storage (swift) - Account Server
openstack-swift-container-updater.service  loaded active running OpenStack Object Storage (swift) - Container Updater
openstack-swift-container.service          loaded active running OpenStack Object Storage (swift) - Container Server
openstack-swift-object-updater.service     loaded active running OpenStack Object Storage (swift) - Object Updater
openstack-swift-object.service             loaded active running OpenStack Object Storage (swift) - Object Server
openstack-swift-proxy.service              loaded active running OpenStack Object Storage (swift) - Proxy Server
openstack-zaqar.service                    loaded active running OpenStack Message Queuing Service (code-named Zaqar) Server
openstack-zaqar@1.service                  loaded active running OpenStack Message Queuing Service (code-named Zaqar) Server Instance 1
openvswitch.service                        loaded active exited  Open vSwitch

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

37 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

备份

1.在执行备份过程之前，请确认您有足够的可用磁盘空间。预计该目标至少为3.5 GB。

[stack@director ~]$df -h

2.以根用户身份运行这些命令，将数据从下云节点备份到名为undercloud-backup-[timestamp].tar.gz的文件。

[root@director ~]# mysqldump --opt --all-databases > /root/undercloud-all-databases.sql 
[root@director ~]# tar --xattrs -czf undercloud-backup-`date +%F`.tar.gz /root/undercloud-all-databases.sql 
/etc/my.cnf.d/server.cnf /var/lib/glance/images /srv/node /home/stack 
tar: Removing leading `/' from member names 

安装新OSPD节点

UCS服务器安装

1.安装新UCS C240 M4服务器的步骤和初始设置步骤可参阅此处：Cisco UCS C240 M4服务器安装和服务指南

2.使用CIMC IP登录服务器。

3.如果固件与之前使用的推荐版本不同，请执行BIOS升级。BIOS升级步骤如下：Cisco UCS C系列机架式服务器BIOS升级指南

5.从RAID级别为1的物理驱动器创建虚拟驱动器。导航到Storage > Cisco 12G SAS模块化RAID控制器(SLOT-HBA)> Controller Info > Create Virtual Drive from Unused Physical Drives（从未使用的物理驱动器创建虚拟驱动器），如此图所示。

6.选择VD并配置“设置为引导驱动器”，如此映像所示。

7.启用IPMI over LAN。导航至Admin > Communication Services > Communication Services，如此图所示。

8.禁用超线程。导航至计算> BIOS >配置BIOS >高级>处理器配置，如此图所示。

 

注意：此处显示的图像和本节中提及的配置步骤均参考固件版本3.0(3e)，如果您使用其他版本，可能会略有变化。

  

Redhat安装

安装Red Hat ISO映像

1.登录OSP-D服务器

2.启动KVM控制台

3.选择“虚拟媒体”>“激活虚拟设备”。接受会话并记住您为未来连接所做的设置。

4.选择“虚拟媒体”>“映射CD/DVD”并映射Red Hat ISO映像。

5.选择“电源”>“重置系统(热启动)”以重新启动系统。

6.重新启动后，按F6并选择Cisco vKVM-Mapped vDVD1.22，然后按Enter。

7.安装RHEL。

恢复云下

根据备份准备云下安装

当计算机安装了RHEL 7.3且处于干净状态后，请重新启用安装和运行指挥交换机所需的所有订用/存储库。

1.主机名配置。

[root@director ~]$sudo hostnamectl set-hostname <FQDN_hostname> 
[root@director ~]$sudo hostnamectl set-hostname --transient <FQDN_hostname>

2.编辑/etc/hosts文件。

[root@director ~]$ vi /etc/hosts

<ospd_external_address>  <server_hostname>  <FQDN_hostname>
10.225.247.142  pod1-ospd pod1-ospd.cisco.com

3.验证主机名。

[root@director ~]$ cat /etc/hostname
pod1-ospd.cisco.com

4.验证DNS配置。

[root@director ~]$ cat /etc/resolv.conf

#Generated by NetworkManager
nameserver <DNS_IP>

5.修改调配NIC接口。

[root@director ~]$ cat /etc/sysconfig/network-scripts/ifcfg-eno1

DEVICE=eno1
ONBOOT=yes
HOTPLUG=no
NM_CONTROLLED=no
PEERDNS=no
DEVICETYPE=ovs
TYPE=OVSPort
OVS_BRIDGE=br-ctlplane
BOOTPROTO=none
MTU=1500

 完成Redhat注册

  

1.下载此软件包，将订用管理器配置为使用rh-satellite。

[root@director ~]$ rpm -Uvh http://
     
     
       /pub/katello-ca-consumer-latest.noarch.rpm 
     
[root@director ~]$ subscription-manager config

2.使用RHEL 7.3的此活动键向rh-satellite注册。 

[root@director ~]$subscription-manager register --org="<ORG>" --activationkey="<KEY>"

3.查看订用。

[root@director ~]$ subscription-manager list –consumed

4.启用与旧OSPD回购相同的存储库。

[root@director ~]$ sudo subscription-manager repos --disable=*
[root@director ~]$ subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-extras-rpms --enable=rh
el-7-server-openstack-10-rpms --enable=rhel-7-server-rh-common-rpms --enable=rhel-ha-for-rhel-7-server-rpm

5.在系统上执行更新以确保您拥有最新的基本系统软件包并重新启动系统。

[root@director ~]$sudo yum update -y
[root@director ~]$sudo reboot

云下恢复

启用订阅后，将云下tar文件下cloud-backup-'date +%F'.tar.gz下备份的tar文件导入到新的OSPD服务器根目录/root。

1.安装mariadb服务器。

[root@director ~]$ yum install -y mariadb-server

2.提取MariaDB配置文件和数据库备份。以根用户身份执行此操作。

[root@director ~]$ tar -xzC / -f undercloud-backup-$DATE.tar.gz etc/my.cnf.d/server.cnf
[root@director ~]$ tar -xzC / -f undercloud-backup-$DATE.tar.gz root/undercloud-all-databases.sql

3.编辑/etc/my.cnf.d/server.cnf并注释掉bind-address条目（如果存在）。

[root@tb3-ospd ~]# vi /etc/my.cnf.d/server.cnf

4.启动MariaDB服务并临时更新max_allowed_packet设置：

[root@director ~]$ systemctl start mariadb
[root@director ~]$ mysql -uroot -e"set global max_allowed_packet = 16777216;"

5.清除某些权限（稍后重新创建）：

[root@director ~]$ for i in ceilometer glance heat ironic keystone neutron nova;do mysql -e "drop user $i";done
[root@director ~]$ mysql -e 'flush privileges'

注意：如果之前在设置中禁用了云顶计服务，请执行以上命令删除“云顶计”。

6.创建堆叠用户帐户。

[root@director ~]$ sudo useradd stack
[root@director ~]$ sudo passwd stack << specify a password
[root@director ~]$ echo "stack ALL=(root) NOPASSWD:ALL" | sudo tee -a /etc/sudoers.d/stack
[root@director ~]$ sudo chmod 0440 /etc/sudoers.d/stack 

7.恢复堆栈用户主目录。

[root@director ~]$ tar -xzC / -f undercloud-backup-$DATE.tar.gz home/stack

8.安装快速、概览基本包，然后恢复其数据。

[root@director ~]$ yum install -y openstack-glance openstack-swift
[root@director ~]$ tar --xattrs -xzC / -f undercloud-backup-$DATE.tar.gz srv/node var/lib/glance/images

9.确认数据归正确用户所有。

[root@director ~]$ chown -R swift: /srv/node
[root@director ~]$ chown -R glance: /var/lib/glance/images

10.恢复云下SSL证书（可选 — 仅在设置使用SSL证书时完成）。

[root@director ~]$ tar -xzC / -f undercloud-backup-$DATE.tar.gz etc/pki/instack-certs/undercloud.pem
[root@director ~]$ tar -xzC / -f undercloud-backup-$DATE.tar.gz etc/pki/ca-trust/source/anchors/ca.crt.pem

11.以stackuser身份重新运行下云安装，确保在堆栈用户主目录中运行它。

[root@director ~]$ su - stack
[stack@director ~]$ sudo yum install -y python-tripleoclient

12.确认主机名在/etc/hosts中设置正确。

13.重新安装下云。

[stack@director ~]$ openstack undercloud install

<snip>
#############################################################################
Undercloud install complete.

The file containing this installation's passwords is at
/home/stack/undercloud-passwords.conf.

There is also a stackrc file at /home/stack/stackrc.

These files are needed to interact with the OpenStack services, and must be
secured.

#############################################################################

将已恢复的下云重新连接到超云

完成这些步骤后，可能会期望下云自动恢复其与超云的连接。节点将继续轮询业务流程（热）以查找待处理任务，使用每几秒发出一次简单HTTP请求。

验证已完成的恢复

请使用这些命令对新恢复的环境执行运行状况检查。

[root@director ~]$ su - stack
Last Log in: Tue Nov 28 21:27:50 EST 2017 from 10.86.255.201 on pts/0

[stack@director ~]$ source stackrc
[stack@director ~]$ nova list
+--------------------------------------+--------------------+--------+------------+-------------+------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------------------+--------+------------+-------------+------------------------+
| b1f5294a-629e-454c-b8a7-d15e21805496 | pod1-compute-0 | ACTIVE | - | Running | ctlplane=192.200.0.119 |
| 9106672e-ac68-423e-89c5-e42f91fefda1 | pod1-compute-1 | ACTIVE | - | Running | ctlplane=192.200.0.120 |
| b3ed4a8f-72d2-4474-91a1-b6b70dd99428 | pod1-compute-2 | ACTIVE | - | Running | ctlplane=192.200.0.124 |
| 677524e4-7211-4571-ac35-004dc5655789 | pod1-compute-3 | ACTIVE | - | Running | ctlplane=192.200.0.107 |
| 55ea7fe5-d797-473c-83b1-d897b76a7520 | pod1-compute-4 | ACTIVE | - | Running | ctlplane=192.200.0.122 |
| c34c1088-d79b-42b6-9306-793a89ae4160 | pod1-compute-5 | ACTIVE | - | Running | ctlplane=192.200.0.108 |
| 4ba28d8c-fb0e-4d7f-8124-77d56199c9b2 | pod1-compute-6 | ACTIVE | - | Running | ctlplane=192.200.0.105 |
| d32f7361-7e73-49b1-a440-fa4db2ac21b1 | pod1-compute-7 | ACTIVE | - | Running | ctlplane=192.200.0.106 |
| 47c6a101-0900-4009-8126-01aaed784ed1 | pod1-compute-8 | ACTIVE | - | Running | ctlplane=192.200.0.121 |
| 1a638081-d407-4240-b9e5-16b47e2ff6a2 | pod1-compute-9 | ACTIVE | - | Running | ctlplane=192.200.0.112 |
<snip>

[stack@director ~]$ ssh heat-admin@192.200.0.107
[heat-admin@pod1-controller-0 ~]$ sudo pcs status

Cluster name: tripleo_cluster
Stack: corosync
Current DC: pod1-controller-0 (version 1.1.15-11.el7_3.4-e174ec8) - partition with quorum

3 nodes and 22 resources configured

Online: [ pod1-controller-0 pod1-controller-1 pod1-controller-2 ]
Full list of resources:

 ip-10.1.10.10 (ocf::heartbeat:IPaddr2): Started pod1-controller-0
 ip-11.120.0.97 (ocf::heartbeat:IPaddr2): Started pod1-controller-1

 Clone Set: haproxy-clone [haproxy]
 Started: [ pod1-controller-0 pod1-controller-1 pod1-controller-2 ]
 Master/Slave Set: galera-master [galera]

 Masters: [ pod1-controller-0 pod1-controller-1 pod1-controller-2 ]

 ip-192.200.0.106 (ocf::heartbeat:IPaddr2): Started pod1-controller-0
 ip-11.120.0.95 (ocf::heartbeat:IPaddr2): Started pod1-controller-1
 ip-11.119.0.98 (ocf::heartbeat:IPaddr2): Started pod1-controller-0
 ip-11.118.0.92 (ocf::heartbeat:IPaddr2): Started pod1-controller-1

 Clone Set: rabbitmq-clone [rabbitmq]
 Started: [ pod1-controller-0 pod1-controller-1 pod1-controller-2 ]
 Master/Slave Set: redis-master [redis]
 Masters: [ pod1-controller-0 ]
 Slaves: [ pod1-controller-1 pod1-controller-2 ]

 openstack-cinder-volume (systemd:openstack-cinder-volume): Started pod1-controller-0
 my-ipmilan-for-controller-0 (stonith:fence_ipmilan): Stopped
 my-ipmilan-for-controller-1 (stonith:fence_ipmilan): Stopped
 my-ipmilan-for-controller-2 (stonith:fence_ipmilan): Stopped

Failed Actions:

* my-ipmilan-for-controller-0_start_0 on pod1-controller-1 'unknown error' (1): call=190, status=Timed Out, exitreason='none',
 last-rc-change='Wed Nov 22 13:52:45 2017', queued=0ms, exec=20005ms

* my-ipmilan-for-controller-1_start_0 on pod1-controller-1 'unknown error' (1): call=192, status=Timed Out, exitreason='none',
 last-rc-change='Wed Nov 22 13:53:08 2017', queued=0ms, exec=20005ms

* my-ipmilan-for-controller-2_start_0 on pod1-controller-1 'unknown error' (1): call=188, status=Timed Out, exitreason='none',
 last-rc-change='Wed Nov 22 13:52:23 2017', queued=0ms, exec=20004ms

* my-ipmilan-for-controller-0_start_0 on pod1-controller-0 'unknown error' (1): call=210, status=Timed Out, exitreason='none',
 last-rc-change='Wed Nov 22 13:53:08 2017', queued=0ms, exec=20005ms

* my-ipmilan-for-controller-1_start_0 on pod1-controller-0 'unknown error' (1): call=207, status=Timed Out, exitreason='none',
 last-rc-change='Wed Nov 22 13:52:45 2017', queued=0ms, exec=20004ms

* my-ipmilan-for-controller-2_start_0 on pod1-controller-0 'unknown error' (1): call=206, status=Timed Out, exitreason='none',
 last-rc-change='Wed Nov 22 13:52:45 2017', queued=0ms, exec=20006ms

* ip-192.200.0.106_monitor_10000 on pod1-controller-0 'not running' (7): call=197, status=complete, exitreason='none',
 last-rc-change='Wed Nov 22 13:51:31 2017', queued=0ms, exec=0ms

* my-ipmilan-for-controller-0_start_0 on pod1-controller-2 'unknown error' (1): call=183, status=Timed Out, exitreason='none',
 last-rc-change='Wed Nov 22 13:52:23 2017', queued=1ms, exec=20006ms

* my-ipmilan-for-controller-1_start_0 on pod1-controller-2 'unknown error' (1): call=184, status=Timed Out, exitreason='none',
 last-rc-change='Wed Nov 22 13:52:23 2017', queued=0ms, exec=20005ms

* my-ipmilan-for-controller-2_start_0 on pod1-controller-2 'unknown error' (1): call=177, status=Timed Out, exitreason='none',
 last-rc-change='Wed Nov 22 13:52:02 2017', queued=0ms, exec=20005ms

Daemon Status:

 corosync: active/enabled
 pacemaker: active/enabled
 pcsd: active/enabled

[heat-admin@pod1-controller-0 ~]$ sudo ceph status

 cluster eb2bb192-b1c9-11e6-9205-525400330666
 health HEALTH_OK
 monmap e1: 3 mons at {pod1-controller-0=11.118.0.40:6789/0,pod1-controller-1=11.118.0.41:6789/0,pod1-controller-2=11.118.0.42:6789/0}
 election epoch 58, quorum 0,1,2 pod1-controller-0,pod1-controller-1,pod1-controller-2
 osdmap e1398: 12 osds: 12 up, 12 in
 flags sortbitwise,require_jewel_osds
 pgmap v1245812: 704 pgs, 6 pools, 542 GB data, 352 kobjects
 1625 GB used, 11767 GB / 13393 GB avail
 704 active+clean
 client io 21549 kB/s wr, 0 op/s rd, 120 op/s wr

检查身份服务(Keystone)操作

此步骤通过查询用户列表来验证身份服务操作。

[stack@director ~]$ source stackrc 
[stack@director ~]$ openstack user list
+----------------------------------+------------------+
| ID | Name |
+----------------------------------+------------------+
| 69ac2b9d89414314b1366590c7336f7d | admin |
| f5c30774fe8f49d0a0d89d5808a4b2cc | glance |
| 3958d852f85749f98cca75f26f43d588 | heat |
| cce8f2b7f1a843a08d0bb295a739bd34 | ironic |
| ce7c642f5b5741b48a84f54d3676b7ee | ironic-inspector |
| a69cd42a5b004ec5bee7b7a0c0612616 | mistral |
| 5355eb161d75464d8476fa0a4198916d | neutron |
| 7cee211da9b947ef9648e8fe979b4396 | nova |
| f73d36563a4a4db482acf7afc7303a32 | swift |
| d15c12621cbc41a8a4b6b67fa4245d03 | zaqar |
| 3f0ed37f95544134a15536b5ca50a3df | zaqar-websocket |
+----------------------------------+------------------+
[stack@director ~]$ 
[stack@director ~]$ source <overcloudrc>
[stack@director ~]$ openstack user list
+----------------------------------+------------+
| ID | Name |
+----------------------------------+------------+
| b4e7954942184e2199cd067dccdd0943 | admin |
| 181878efb6044116a1768df350d95886 | neutron |
| 6e443967ee3f4943895c809dc998b482 | heat |
| c1407de17f5446de821168789ab57449 | nova |
| c9f64c5a2b6e4d4a9ff6b82adef43992 | glance |
| 800e6b1163b74cc2a5fab4afb382f37d | cinder |
| 4cfa5a2a44c44c678025842f080e5f53 | heat-cfn |
| 9b222eeb8a58459bb3bfc76b8fff0f9f | swift |
| 815f3f25bcda49c290e1b56cd7981d1b | core |
| 07c40ade64f34a64932129175150fa4a | gnocchi |
| 0ceeda0bc32c4d46890e53adef9a193d | aodh |
| f3caab060171468592eab376a94967b8 | ceilometer |
+----------------------------------+------------+
[stack@director ~]$ 

上传映像以供将来节点内省

验证/httpboot所有这些文件inspector.ipxe、agent.kernel、agent.ramdisk。否则，请继续执行以下步骤以在映像中更新。

[stack@director ~]$ ls /httpboot
inspector.ipxe
[stack@director ~]$ source stackrc 
[stack@director ~]$ cd images/
[stack@director images]$ openstack overcloud image upload --image-path /home/stack/images
Image "overcloud-full-vmlinuz" is up-to-date, skipping.
Image "overcloud-full-initrd" is up-to-date, skipping.
Image "overcloud-full" is up-to-date, skipping.
Image "bm-deploy-kernel" is up-to-date, skipping.
Image "bm-deploy-ramdisk" is up-to-date, skipping.
[stack@director images]$ ls /httpboot
agent.kernel agent.ramdisk inspector.ipxe
[stack@director images]$ 

重新启动围栏

OSPD恢复后，击剑将处于停止状态。此过程将启用隔离。

[heat-admin@pod1-controller-0 ~]$ sudo pcs property set stonith-enabled=true
[heat-admin@pod1-controller-0 ~]$ sudo pcs status
[heat-admin@pod1-controller-0 ~]$sudo pcs stonith show 

相关信息

• 备份和恢复Director Undercloud

• 隔离控制器节点

• 技术支持和文档 - Cisco Systems