更換OSPD伺服器UCS 240M4 - CPAR

簡介

本文說明在Ultra-M安裝中更換託管OpenStack平台導向器(OSPD)的故障伺服器所需的步驟。 此過程適用於使用NEWTON版本的OpenStack環境，其中ESC不管理Cisco Prime Access Registrar(CPAR)，而CPAR直接安裝在OpenStack上部署的VM上。

背景資訊

Ultra-M是經過預打包和驗證的虛擬化移動資料包核心解決方案，旨在簡化VNF的部署。 OpenStack是適用於Ultra-M的虛擬化基礎架構管理器(VIM)，包含以下節點型別：

• 計算
• 對象儲存磁碟 — 計算（OSD — 計算）
• 控制器
• OSPD

Ultra-M的高級體系結構及涉及的元件如下圖所示：

本文檔面向熟悉Cisco Ultra-M平台的思科人員，詳細說明了在OpenStack和Redhat作業系統上需要執行的步驟。 

---

附註：Ultra M 5.1.x版本用於定義本文檔中的過程。

---

縮寫

澳門幣	過程方法
OSD	對象儲存磁碟
OSPD	OpenStack平台導向器
硬碟	硬碟驅動器
固態硬碟	固態驅動器
VIM	虛擬基礎架構管理員
虛擬機器	虛擬機器
EM	元素管理器
UAS	Ultra自動化服務
UUID	通用唯一識別符號

MoP的工作流程

替換過程的高級工作流

必要條件

狀態檢查

在更換OSPD伺服器之前，請務必檢查Red Hat OpenStack平台環境的當前狀態，並確保其處於正常狀態，以避免更換過程啟動時出現問題。 

1.檢查OpenStack堆疊的狀態和節點清單：

[stack@director ~]$ source stackrc 
[stack@director ~]$ openstack stack list --nested
[stack@director ~]$ ironic node-list
[stack@director ~]$ nova list

2.從OSP-D節點檢查所有底層雲服務是否處於已載入、活動和運行狀態：

[stack@al03-pod2-ospd ~]$ systemctl list-units "openstack*" "neutron*" "openvswitch*"
 UNIT LOAD ACTIVE SUB DESCRIPTION
 neutron-dhcp-agent.service loaded active running OpenStack Neutron DHCP Agent
 neutron-metadata-agent.service loaded active running OpenStack Neutron Metadata Agent
 neutron-openvswitch-agent.service loaded active running OpenStack Neutron Open vSwitch Agent
 neutron-server.service loaded active running OpenStack Neutron Server
 openstack-aodh-evaluator.service loaded active running OpenStack Alarm evaluator service
 openstack-aodh-listener.service loaded active running OpenStack Alarm listener service
 openstack-aodh-notifier.service loaded active running OpenStack Alarm notifier service
 openstack-ceilometer-central.service loaded active running OpenStack ceilometer central agent
 openstack-ceilometer-collector.service loaded active running OpenStack ceilometer collection service
 openstack-ceilometer-notification.service loaded active running OpenStack ceilometer notification agent
 openstack-glance-api.service loaded active running OpenStack Image Service (code-named Glance) API server
 openstack-glance-registry.service loaded active running OpenStack Image Service (code-named Glance) Registry server
 openstack-heat-api-cfn.service loaded active running Openstack Heat CFN-compatible API Service
 openstack-heat-api.service loaded active running OpenStack Heat API Service
 openstack-heat-engine.service loaded active running Openstack Heat Engine Service
 openstack-ironic-api.service loaded active running OpenStack Ironic API service
 openstack-ironic-conductor.service loaded active running OpenStack Ironic Conductor service
 openstack-ironic-inspector-dnsmasq.service loaded active running PXE boot dnsmasq service for Ironic Inspector
 openstack-ironic-inspector.service loaded active running Hardware introspection service for OpenStack Ironic
 openstack-mistral-api.service loaded active running Mistral API Server
 openstack-mistral-engine.service loaded active running Mistral Engine Server
 openstack-mistral-executor.service loaded active running Mistral Executor Server
 openstack-nova-api.service loaded active running OpenStack Nova API Server
 openstack-nova-cert.service loaded active running OpenStack Nova Cert Server
 openstack-nova-compute.service loaded active running OpenStack Nova Compute Server
 openstack-nova-conductor.service loaded active running OpenStack Nova Conductor Server
 openstack-nova-scheduler.service loaded active running OpenStack Nova Scheduler Server
 openstack-swift-account-reaper.service loaded active running OpenStack Object Storage (swift) - Account Reaper
 openstack-swift-account.service loaded active running OpenStack Object Storage (swift) - Account Server
 openstack-swift-container-updater.service loaded active running OpenStack Object Storage (swift) - Container Updater
 openstack-swift-container.service loaded active running OpenStack Object Storage (swift) - Container Server
 openstack-swift-object-updater.service loaded active running OpenStack Object Storage (swift) - Object Updater
 openstack-swift-object.service loaded active running OpenStack Object Storage (swift) - Object Server
 openstack-swift-proxy.service loaded active running OpenStack Object Storage (swift) - Proxy Server
 openstack-zaqar.service loaded active running OpenStack Message Queuing Service (code-named Zaqar) Server
 openstack-zaqar@1.service loaded active running OpenStack Message Queuing Service (code-named Zaqar) Server Instance 1
 openvswitch.service loaded active exited Open vSwitch

LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
lines 1-43 
lines 2-44 37 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
lines 4-46/46 (END) lines 4-46/46 (END) lines 4-46/46 (END) lines 4-46/46 (END) lines 4-46/46 (END)

備份

1.在執行備份過程之前，請確認是否有足夠的可用磁碟空間。此彈珠應至少為3.5 GB。

[stack@director ~]$ df -h

2.以root使用者身份運行這些命令，以便將資料從底層雲節點備份到名為undercloud-backup-[timestamp].tar.gz的檔案中。

[root@director ~]# mysqldump --opt --all-databases > /root/undercloud-all-databases.sql
[root@director ~]# tar --xattrs -czf undercloud-backup-`date +%F`.tar.gz /root/undercloud-all-databases.sql 
/etc/my.cnf.d/server.cnf /var/lib/glance/images /srv/node /home/stack
tar: Removing leading `/' from member names

安裝新的OSPD節點

UCS伺服器安裝

1.安裝新UCS C240 M4伺服器的步驟和初始設定步驟可從Cisco UCS C240 M4伺服器安裝和服務指南中參考。

2.安裝伺服器後，將硬碟插入其各自的插槽中作為舊伺服器。

3.使用思科整合式管理控制器(CIMC)IP登入伺服器。

4.如果韌體與以前使用的推薦版本不一致，請執行BIOS升級。此處提供了BIOS升級步驟：Cisco UCS C系列機架式伺服器BIOS升級指南。

5.檢驗物理驅動器的狀態。它必須是未配置Good。導覽至Storage > Cisco 12G SAS Modular Raid Controller(SLOT-HBA)> Physical Drive Info，如下圖所示。

6.從RAID級別為1的物理驅動器建立虛擬驅動器。導航到Storage > Cisco 12G SAS Modular Raid Controller(SLOT-HBA)> Controller Info > Create Virtual Drive from Unused Physical Drives，如下圖所示。

7.選擇VD並配置Set as Boot Drive，如下圖所示。

8.啟用IPMI over LAN。導覽至Admin > Communication Services > Communication Services，如下圖所示。

  

9.禁用超執行緒。導覽至Compute > BIOS > Configure BIOS > Advanced > Processor Configuration，如下圖所示。

---

附註：此處顯示的影象和本節中提到的配置步驟是參考韌體版本3.0(3e)，如果您使用其他版本，可能會有細微的變化。

---

Redhat安裝

安裝Red Hat ISO映像

1.登入OSP-D伺服器。

2.啟動KVM控制檯。

3.導航至 虛擬媒體>啟用虛擬裝置.接受會話並啟用Remembering your Setting for Future Connections。

4.導航至 虛擬媒體> 對映CD/DVD並對映Red Hat ISO映像。

5.導航至 電源> 按順序重置系統(熱啟動) 重新啟動系統。

6.重新啟動時，請按 F6並選擇Cisco vKVM對映vDVD1.22並按Enter鍵。

安裝RHEL

根據備份準備雲下安裝

在電腦安裝有RHEL 7.3並處於乾淨狀態後，重新啟用安裝和運行director所需的所有訂閱/儲存庫。

1.主機名配置。

[root@director ~]$sudo hostnamectl set-hostname <FQDN_hostname> 
[root@director ~]$sudo hostnamectl set-hostname --transient <FQDN_hostname>

2.編輯/etc/hosts文件。

[root@director ~]$ vi /etc/hosts
<ospd_external_address>  <server_hostname>  <FQDN_hostname>
192.168.247.142  pod2-stack-ospd pod2-stack-ospd.cisco.com

3.驗證主機名。

[root@director ~]$ cat /etc/hostname
pod2-stack-ospd.cisco.com

4.驗證DNS配置。

[root@director ~]$ cat /etc/resolv.conf

#Generated by NetworkManager
nameserver <DNS_IP>

5.修改調配nic介面。

[root@director ~]$ cat /etc/sysconfig/network-scripts/ifcfg-eno1

DEVICE=eno1
ONBOOT=yes
HOTPLUG=no
NM_CONTROLLED=no
PEERDNS=no
DEVICETYPE=ovs
TYPE=OVSPort
OVS_BRIDGE=br-ctlplane
BOOTPROTO=none
MTU=1500

完成Redhat註冊

  

1.下載此軟體包，以便將subscription-manager配置為使用rh-satellite。

[root@director ~]$ rpm -Uvh http://
     
     
       /pub/katello-ca-consumer-latest.noarch.rpm 
     
[root@director ~]$ subscription-manager config

2.使用此針對RHEL 7.3的啟用金鑰向rh衛星註冊。 

[root@director ~]$subscription-manager register --org="<ORG>" --activationkey="<KEY>"

3.要檢視訂用，請執行以下操作：

[root@director ~]$ subscription-manager list –consumed

4.啟用與舊的OSPD重定位相同的儲存庫。

[root@director ~]$ sudo subscription-manager repos --disable=*
[root@director ~]$ subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-extras-rpms --enable=rh
el-7-server-openstack-10-rpms --enable=rhel-7-server-rh-common-rpms --enable=rhel-ha-for-rhel-7-server-rpm

5.對系統進行更新，以確保您擁有最新的基本系統軟體包並重新啟動系統。

[root@director ~]$sudo yum update -y
[root@director ~]$sudo reboot

Undercloud Restoration

啟用訂閱後，將備份的undercloud tar檔案「undercloud-backup-'date +%F'.tar.gz」匯入到新的OSP-D伺服器根目錄/root

1.安裝mariadb伺服器。

[root@director ~]$ yum install -y mariadb-server

2.提取MariaDB配置檔案和資料庫備份。以root使用者身份執行此操作。

[root@director ~]$ tar -xzC / -f undercloud-backup-$DATE.tar.gz etc/my.cnf.d/server.cnf
[root@director ~]$ tar -xzC / -f undercloud-backup-$DATE.tar.gz root/undercloud-all-databases.sql

3.編輯/etc/my.cnf.d/server.cnf，註釋掉bind-address條目（如果存在）。

[root@tb3-ospd ~]# vi /etc/my.cnf.d/server.cnf

4.啟動MariaDB服務並暫時更新max_allowed_packet設定。

[root@director ~]$ systemctl start mariadb
[root@director ~]$ mysql -uroot -e"set global max_allowed_packet = 16777216;"

5.清除某些許可權（稍後重新建立）。

[root@director ~]$ for i in ceilometer glance heat ironic keystone neutron nova;do mysql -e "drop user $i";done
[root@director ~]$ mysql -e 'flush privileges'

---

註：如果設定中以前禁用了雲器服務，請執行上述命令刪除「雲器」。

---

6.建立stackuser帳戶。

[root@director ~]$ sudo useradd stack
[root@director ~]$ sudo passwd stack << specify a password
[root@director ~]$ echo "stack ALL=(root) NOPASSWD:ALL" | sudo tee -a /etc/sudoers.d/stack
[root@director ~]$ sudo chmod 0440 /etc/sudoers.d/stack 

7.還原堆疊使用者主目錄。

[root@director ~]$ tar -xzC / -f undercloud-backup-$DATE.tar.gz home/stack

8.安裝swift和glance base程式包，然後恢復其資料。

[root@director ~]$ yum install -y openstack-glance openstack-swift
[root@director ~]$ tar --xattrs -xzC / -f undercloud-backup-$DATE.tar.gz srv/node var/lib/glance/images

9.確認資料為正確使用者所有。

[root@director ~]$ chown -R swift: /srv/node
[root@director ~]$ chown -R glance: /var/lib/glance/images

10.還原雲下SSL證書（可選 — 僅在安裝程式使用SSL證書時完成）。

[root@director ~]$ tar -xzC / -f undercloud-backup-$DATE.tar.gz etc/pki/instack-certs/undercloud.pem
[root@director ~]$ tar -xzC / -f undercloud-backup-$DATE.tar.gz etc/pki/ca-trust/source/anchors/ca.crt.pem

11.以stackuser身份重新運行undercloud安裝，確保在堆疊使用者主目錄中運行該安裝：

[root@director ~]$ su - stack
[stack@director ~]$ sudo yum install -y python-tripleoclient

12.確認已在/etc/hosts中正確設定主機名。

13.重新安裝雲下。

[stack@director ~]$ openstack undercloud install

<snip>
#############################################################################
Undercloud install complete.

The file containing this installation's passwords is at
/home/stack/undercloud-passwords.conf.

There is also a stackrc file at /home/stack/stackrc.

These files are needed to interact with the OpenStack services, and must be
secured.

#############################################################################

將已恢復的Undercloud重新連線到Overcloud

完成上述步驟後，可以預期底層雲自動恢復其與上層雲的連線。節點將繼續使用每隔幾秒發出的簡單HTTP請求輪詢協調（加熱）以查詢待執行任務。

驗證已完成還原

使用這些命令對新恢復的環境執行運行狀況檢查。

[root@director ~]$ su - stack
Last Log in: Tue Nov 28 21:27:50 EST 2017 from 192.182.255.20 on pts/0

[stack@director ~]$ source stackrc
[stack@director ~]$ nova list
+--------------------------------------+--------------------------+--------+------------+-------------+------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------------------------+--------+------------+-------------+------------------------+
| 03f15071-21aa-4bcf-8fdd-acdbde305168 | pod2-stack-compute-0 | ACTIVE | - | Running | ctlplane=192.200.0.106 |
| 1f725ce3-948d-49e9-aed9-b99e73d82644 | pod2-stack-compute-1 | ACTIVE | - | Running | ctlplane=192.200.0.107 |
| fbc13c78-dc06-4ac9-a3c5-595ccc147adc | pod2-stack-compute-2 | ACTIVE | - | Running | ctlplane=192.200.0.119 |
| 3b94e0b1-47dc-4960-b3eb-d02ffe9ae693 | pod2-stack-compute-3 | ACTIVE | - | Running | ctlplane=192.200.0.112 |
| 5dbac94d-19b9-493e-a366-1e2e2e5e34c5 | pod2-stack-compute-4 | ACTIVE | - | Running | ctlplane=192.200.0.116 |
| b896c73f-d2c8-439c-bc02-7b0a2526dd70 | pod2-stack-controller-0 | ACTIVE | - | Running | ctlplane=192.200.0.113 |
| 2519ce67-d836-4e5f-a672-1a915df75c7c | pod2-stack-controller-1 | ACTIVE | - | Running | ctlplane=192.200.0.105 |
| e19b9625-5635-4a52-a369-44310f3e6a21 | pod2-stack-controller-2 | ACTIVE | - | Running | ctlplane=192.200.0.120 |
| 6810c884-1cb9-4321-9a07-192443920f1f | pod2-stack-osd-compute-0 | ACTIVE | - | Running | ctlplane=192.200.0.109 |
| 26d3f7b1-ba97-431f-aa6e-ba91661db45d | pod2-stack-osd-compute-1 | ACTIVE | - | Running | ctlplane=192.200.0.117 |
| 6e4a8aa9-4870-465a-a7e2-0932ff55e34b | pod2-stack-osd-compute-2 | ACTIVE | - | Running | ctlplane=192.200.0.103 |
+--------------------------------------+--------------------------+--------+------------+-------------+------------------------+
<snip>

[stack@director ~]$ ssh heat-admin@192.200.0.113
Last login: Fri Jul 6 09:02:41 2018 from 192.200.0.1 
[heat-admin@pod2-stack-controller-0 ~]$ sudo pcs status
Cluster name: tripleo_cluster
Stack: corosync
Current DC: pod2-stack-controller-2 (version 1.1.15-11.el7_3.4-e174ec8) - partition with quorum
Last updated: Tue Jul 10 10:04:15 2018Last change: Fri Jul 6 09:03:35 2018 by root via crm_attribute on pod2-stack-controller-0

3 nodes and 19 resources configured

Online: [ pod2-stack-controller-0 pod2-stack-controller-1 pod2-stack-controller-2 ]

Full list of resources:

 ip-11.120.0.49(ocf::heartbeat:IPaddr2):Started pod2-stack-controller-1
 Clone Set: haproxy-clone [haproxy]
 Started: [ pod2-stack-controller-0 pod2-stack-controller-1 pod2-stack-controller-2 ]
 Master/Slave Set: galera-master [galera]
 Masters: [ pod2-stack-controller-0 pod2-stack-controller-1 pod2-stack-controller-2 ]
 ip-192.200.0.110(ocf::heartbeat:IPaddr2):Started pod2-stack-controller-1
 ip-11.120.0.44(ocf::heartbeat:IPaddr2):Started pod2-stack-controller-2
 ip-11.118.0.49(ocf::heartbeat:IPaddr2):Started pod2-stack-controller-2
 Clone Set: rabbitmq-clone [rabbitmq]
 Started: [ pod2-stack-controller-0 pod2-stack-controller-1 pod2-stack-controller-2 ]
 ip-10.225.247.214(ocf::heartbeat:IPaddr2):Started pod2-stack-controller-1
 Master/Slave Set: redis-master [redis]
 Masters: [ pod2-stack-controller-2 ]
 Slaves: [ pod2-stack-controller-0 pod2-stack-controller-1 ]
 ip-11.119.0.49(ocf::heartbeat:IPaddr2):Started pod2-stack-controller-2
 openstack-cinder-volume(systemd:openstack-cinder-volume):Started pod2-stack-controller-1

Daemon Status:
 corosync: active/enabled
 pacemaker: active/enabled
 pcsd: active/enabled

[heat-admin@pod2-stack-controller-0 ~]$ sudo ceph status
 cluster eb2bb192-b1c9-11e6-9205-525400330666
 health HEALTH_OK
 monmap e1: 3 mons at {pod2-stack-controller-0=11.118.0.10:6789/0,pod2-stack-controller-1=11.118.0.11:6789/0,pod2-stack-controller-2=11.118.0.12:6789/0}
 election epoch 10, quorum 0,1,2 pod2-stack-controller-0,pod2-stack-controller-1,pod2-stack-controller-2
 osdmap e81: 12 osds: 12 up, 12 in
 flags sortbitwise,require_jewel_osds
 pgmap v23094282: 704 pgs, 6 pools, 809 GB data, 424 kobjects
 2418 GB used, 10974 GB / 13393 GB avail
 704 active+clean
 client io 1841 kB/s wr, 0 op/s rd, 220 op/s wr
[heat-admin@pod2-stack-controller-0 ~]$ exit
logout
Connection to 192.200.0.113 closed.

檢查身份服務(Keystone)操作

此步驟通過查詢使用者清單來驗證Identity Service操作。

[stack@director ~]$ source stackerc
[stack@director~]$ openstack user list
+----------------------------------+------------------+
| ID | Name |
+----------------------------------+------------------+
| 4d93cc08ed314b87b4c6dce95c19558f | admin |
| c499bd1a65d94029804350e3bca19888 | aodh |
| d9b59136b2794ec1b9c18c4f7b95c674 | ceilometer |
| 4650cdca79b44f2b9d2426b7a9df9495 | glance |
| 4a6724a623f4463c971658462b505c8a | heat |
| e18ae9a5b7f14aefb9e93931dcec476e | ironic |
| 9831160669c44c10b04d6f78b7517a55 | ironic-inspector |
| 6a5d6f02b9a74dfb9f679eecfebb28e0 | mistral |
| 462c8ec91fb0485fbc7dfec6d4ecf480 | neutron |
| f3a66d84e39c47a0bdcd14d24bd72816 | nova |
| 1f16ea9035ab424a88d5f34681d3f893 | swift |
| ab4a1e5f62be4609b290c32dc93bfbce | zaqar |
| fac68a166f5f442284f7d06355369c53 | zaqar-websocket |
+----------------------------------+------------------+

[stack@director ~]$ source <overcloudrc>
[stack@director ~]$ openstack user list
+----------------------------------+------------+
| ID | Name |
+----------------------------------+------------+
| 7285a65e0c8c4447b10875d0e9c9da69 | admin |
| c7c78ac31ec24926b3eed79e7f5bd59e | neutron |
| 191c2a293beb48edb4d2c1160c9a0523 | heat |
| 1445a6ef2ae94b6f9365ffbc6b56a231 | gnocchi |
| fb8b006129554c6b8465310fcfc0ba8c | aodh |
| 5afcb816f93b4f20998c9edaf3b143a0 | nova |
| 37444af1295344809eca6058a782f871 | glance |
| e74d2d9e59f34ddfb4408a8beb88db7b | ceilometer |
| 9baab90178c141478ce7c0084e930f40 | cinder |
| 5a964c651c11402284eecb0e8f40ee33 | heat-cfn |
| 6d49ebae55da432580943d6080efcf90 | swift |
| 41a666fe64c64df593d73b4f467885be | core |
| ab004ba966bf4ddc92783fbb6045e48b | cpar |
| fce5b5c5dc7647ccbc2170b56a64db33 | cpardemo |
| 28ead2dab0ef486ba449b441715a3e04 | pcrf |
| 0aac4107e3d443dc93ca62c3928a1998 | pcrfdemo |
+----------------------------------+------------+

上傳影象供未來節點內省

1.驗證/httpboot所有這些檔案「inspector.ipxe， agent.kernel， agent.ramdisk」（如果不進行以下步驟以更新映像）。

[stack@director ~]$ ls /httpboot
inspector.ipxe
[stack@director ~]$ source stackrc 
[stack@director ~]$ cd images/
[stack@director images]$ openstack overcloud image upload --image-path /home/stack/images
Image "overcloud-full-vmlinuz" is up-to-date, skipping.
Image "overcloud-full-initrd" is up-to-date, skipping.
Image "overcloud-full" is up-to-date, skipping.
Image "bm-deploy-kernel" is up-to-date, skipping.
Image "bm-deploy-ramdisk" is up-to-date, skipping.
[stack@director images]$ ls /httpboot
agent.kernel agent.ramdisk inspector.ipxe
[stack@director images]$ 

重新啟動圍欄

在OSPD恢復後，圍欄將處於停止狀態，低於程式將啟用圍欄。 

[heat-admin@pod2-stack-controller-0 ~]$ sudo pcs property set stonith-enabled=true
[heat-admin@pod2-stack-controller-0 ~]$ sudo pcs status
[heat-admin@pod2-stack-controller-0 ~]$sudo pcs stonith show 

相關資訊

• 備份和還原Director Undercloud

• 隔離控制器節點

• 技術支援與文件 - Cisco Systems