Replacement of OSPD Server UCS 240M4 - CPAR

Introduction

This document describes the steps required in order to replace a faulty server that hosts the OpenStack Platform Director (OSPD) in an Ultra-M setup. This procedure applies for an OpenStack environment with the use of the NEWTON version where ESC does not manage Cisco Prime Access Registrar (CPAR) and CPAR is installed directly on the VM deployed on OpenStack.

Background Information

Ultra-M is a pre-packaged and validated virtualized mobile packet core solution that is designed in order to simplify the deployment of VNFs. OpenStack is the Virtualized Infrastructure Manager (VIM) for Ultra-M and consists of these node types:

• Compute
• Object Storage Disk - Compute (OSD - Compute)
• Controller
• OSPD

The high-level architecture of Ultra-M and the components involved are depicted in this image:

This document is intended for Cisco personnel who are familiar with Cisco Ultra-M platform and it details the steps that are required to be carried out at OpenStack and Redhat OS. 

Note: Ultra M 5.1.x release is considered in order to define the procedures in this document.

Abbreviations

Workflow of the MoP

High level workflow of the replacement procedure

Prerequisites

Status Check

Before you replace an OSPD server, it is important to check the current state of the Red Hat OpenStack Platform environment and ensure that it is healthy in order to avoid complications when the replacement process is on. 

1. Check the status of OpenStack stack and the node list:

[stack@director ~]$ source stackrc 
[stack@director ~]$ openstack stack list --nested
[stack@director ~]$ ironic node-list
[stack@director ~]$ nova list

2. Check if all the undercloud services are in loaded, active and running status from the OSP-D node:

[stack@al03-pod2-ospd ~]$ systemctl list-units "openstack*" "neutron*" "openvswitch*"
 UNIT LOAD ACTIVE SUB DESCRIPTION
 neutron-dhcp-agent.service loaded active running OpenStack Neutron DHCP Agent
 neutron-metadata-agent.service loaded active running OpenStack Neutron Metadata Agent
 neutron-openvswitch-agent.service loaded active running OpenStack Neutron Open vSwitch Agent
 neutron-server.service loaded active running OpenStack Neutron Server
 openstack-aodh-evaluator.service loaded active running OpenStack Alarm evaluator service
 openstack-aodh-listener.service loaded active running OpenStack Alarm listener service
 openstack-aodh-notifier.service loaded active running OpenStack Alarm notifier service
 openstack-ceilometer-central.service loaded active running OpenStack ceilometer central agent
 openstack-ceilometer-collector.service loaded active running OpenStack ceilometer collection service
 openstack-ceilometer-notification.service loaded active running OpenStack ceilometer notification agent
 openstack-glance-api.service loaded active running OpenStack Image Service (code-named Glance) API server
 openstack-glance-registry.service loaded active running OpenStack Image Service (code-named Glance) Registry server
 openstack-heat-api-cfn.service loaded active running Openstack Heat CFN-compatible API Service
 openstack-heat-api.service loaded active running OpenStack Heat API Service
 openstack-heat-engine.service loaded active running Openstack Heat Engine Service
 openstack-ironic-api.service loaded active running OpenStack Ironic API service
 openstack-ironic-conductor.service loaded active running OpenStack Ironic Conductor service
 openstack-ironic-inspector-dnsmasq.service loaded active running PXE boot dnsmasq service for Ironic Inspector
 openstack-ironic-inspector.service loaded active running Hardware introspection service for OpenStack Ironic
 openstack-mistral-api.service loaded active running Mistral API Server
 openstack-mistral-engine.service loaded active running Mistral Engine Server
 openstack-mistral-executor.service loaded active running Mistral Executor Server
 openstack-nova-api.service loaded active running OpenStack Nova API Server
 openstack-nova-cert.service loaded active running OpenStack Nova Cert Server
 openstack-nova-compute.service loaded active running OpenStack Nova Compute Server
 openstack-nova-conductor.service loaded active running OpenStack Nova Conductor Server
 openstack-nova-scheduler.service loaded active running OpenStack Nova Scheduler Server
 openstack-swift-account-reaper.service loaded active running OpenStack Object Storage (swift) - Account Reaper
 openstack-swift-account.service loaded active running OpenStack Object Storage (swift) - Account Server
 openstack-swift-container-updater.service loaded active running OpenStack Object Storage (swift) - Container Updater
 openstack-swift-container.service loaded active running OpenStack Object Storage (swift) - Container Server
 openstack-swift-object-updater.service loaded active running OpenStack Object Storage (swift) - Object Updater
 openstack-swift-object.service loaded active running OpenStack Object Storage (swift) - Object Server
 openstack-swift-proxy.service loaded active running OpenStack Object Storage (swift) - Proxy Server
 openstack-zaqar.service loaded active running OpenStack Message Queuing Service (code-named Zaqar) Server
 openstack-zaqar@1.service loaded active running OpenStack Message Queuing Service (code-named Zaqar) Server Instance 1
 openvswitch.service loaded active exited Open vSwitch

LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
lines 1-43 
lines 2-44 37 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
lines 4-46/46 (END) lines 4-46/46 (END) lines 4-46/46 (END) lines 4-46/46 (END) lines 4-46/46 (END)

Backup

1. Confirm that you have sufficient disk space available before you perform the backup process. This tarball is expected to be at least 3.5 GB.

[stack@director ~]$ df -h

2. Run these commands as the root user in order to backup the data from the undercloud node to a file named undercloud-backup-[timestamp].tar.gz.

[root@director ~]# mysqldump --opt --all-databases > /root/undercloud-all-databases.sql
[root@director ~]# tar --xattrs -czf undercloud-backup-`date +%F`.tar.gz /root/undercloud-all-databases.sql 
/etc/my.cnf.d/server.cnf /var/lib/glance/images /srv/node /home/stack
tar: Removing leading `/' from member names

Install the New OSPD Node

UCS Server Installation

1. The steps in order to install a new UCS C240 M4 server and the initial setup steps can be referred from Cisco UCS C240 M4 Server Installation and Service Guide.

2. After the installation of the server, insert the hard disks in their respective slots as the old server.

3. Log in to the server with the use of the Cisco Integrated Management Controller (CIMC) IP.

4. Perform BIOS upgrade if the firmware is not as per the recommended version used previously. Steps for BIOS upgrade are given here: Cisco UCS C-Series Rack-Mount Server BIOS Upgrade Guide.

5. Verify the status of the Physical Drives. It must be Unconfigured Good. Navigate to Storage > Cisco 12G SAS Modular Raid Controller (SLOT-HBA) > Physical Drive Info as shown in the image.

6. Create a virtual drive from the physical drives with RAID Level 1. Navigate to Storage > Cisco 12G SAS Modular Raid Controller (SLOT-HBA) > Controller Info > Create Virtual Drive from Unused Physical Drives as shown in the image.

7. Select the VD and configure Set as Boot Drive as shown in the image.

8. Enable IPMI over LAN. Navigate to Admin > Communication Services > Communication Services as shown in the image.

9. Disable hyperthreading. Navigate to Compute > BIOS > Configure BIOS > Advanced > Processor Configuration as shown in the image.

Note: The image shown here and the configuration steps mentioned in this section are with reference to the firmware version 3.0(3e) and there might be slight variations if you work on other versions.

Redhat Installation

Mount the Red Hat ISO Image

1. Log in to the OSP-D Server.

2. Launch KVM Console.

3. Navigate to Virtual Media > Activate Virtual Devices. Accept the session and enable Remembering your Setting for Future Connections.

4. Navigate to Virtual Media > Map CD/DVDand map the Red Hat ISO image.

5. Navigate to Power > Reset System (Warm Boot) in order to reboot the system.

6. Upon restart, press F6and selectCisco vKVM-Mapped vDVD1.22and press Enter.

Install RHEL

Prepare Undercloud Installation Based on Backup

Once the machine is installed with RHEL 7.3 and is in a clean state, re-enable all the subscriptions/repositories that are needed to install and run director.

1. Hostname Configuration.

[root@director ~]$sudo hostnamectl set-hostname <FQDN_hostname> 
[root@director ~]$sudo hostnamectl set-hostname --transient <FQDN_hostname>

2. Edit /etc/hosts file.

[root@director ~]$ vi /etc/hosts
<ospd_external_address>  <server_hostname>  <FQDN_hostname>
192.168.247.142  pod2-stack-ospd pod2-stack-ospd.cisco.com

3. Validate hostname.

[root@director ~]$ cat /etc/hostname
pod2-stack-ospd.cisco.com

4. Validate DNS configuration.

[root@director ~]$ cat /etc/resolv.conf

#Generated by NetworkManager
nameserver <DNS_IP>

5. Modify provisioning nic interface.

[root@director ~]$ cat /etc/sysconfig/network-scripts/ifcfg-eno1

DEVICE=eno1
ONBOOT=yes
HOTPLUG=no
NM_CONTROLLED=no
PEERDNS=no
DEVICETYPE=ovs
TYPE=OVSPort
OVS_BRIDGE=br-ctlplane
BOOTPROTO=none
MTU=1500

Complete the Redhat Registration

1. Download this package in order to configure subscription-manager to use rh-satellite.

[root@director ~]$ rpm -Uvh http://<satellite-server>/pub/katello-ca-consumer-latest.noarch.rpm
[root@director ~]$ subscription-manager config

2. Register with rh-satellite using this activationkey for RHEL 7.3. 

[root@director ~]$subscription-manager register --org="<ORG>" --activationkey="<KEY>"

3. In order to see the subscription.

[root@director ~]$ subscription-manager list –consumed

4. Enable the repositories same as old OSPD repos.

[root@director ~]$ sudo subscription-manager repos --disable=*
[root@director ~]$ subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-extras-rpms --enable=rh
el-7-server-openstack-10-rpms --enable=rhel-7-server-rh-common-rpms --enable=rhel-ha-for-rhel-7-server-rpm

5. Perform an update on your system to Ensure you have the latest base system packages and reboot the system.

[root@director ~]$sudo yum update -y
[root@director ~]$sudo reboot

Undercloud Restoration

After you enable the subscription, import the backed up undercloud tar file “undercloud-backup-`date +%F`.tar.gz” to new OSP-D server root directory /root

1. Install the mariadb server.

[root@director ~]$ yum install -y mariadb-server

2. Extract the MariaDB configuration file and Database backup. Do this operation as root user.

[root@director ~]$ tar -xzC / -f undercloud-backup-$DATE.tar.gz etc/my.cnf.d/server.cnf
[root@director ~]$ tar -xzC / -f undercloud-backup-$DATE.tar.gz root/undercloud-all-databases.sql

3. Edit /etc/my.cnf.d/server.cnf and comment out the bind-address entry if present.

[root@tb3-ospd ~]# vi /etc/my.cnf.d/server.cnf

4. Start the MariaDB service and temporarily update the max_allowed_packet setting.

[root@director ~]$ systemctl start mariadb
[root@director ~]$ mysql -uroot -e"set global max_allowed_packet = 16777216;"

5. Clean up certain permissions (to be recreated later).

[root@director ~]$ for i in ceilometer glance heat ironic keystone neutron nova;do mysql -e "drop user $i";done
[root@director ~]$ mysql -e 'flush privileges'

Note: If ceilometer service has been previously disabled in the setup, execute the above command removing "ceilometer".

6. Create the stackuser account.

[root@director ~]$ sudo useradd stack
[root@director ~]$ sudo passwd stack << specify a password
[root@director ~]$ echo "stack ALL=(root) NOPASSWD:ALL" | sudo tee -a /etc/sudoers.d/stack
[root@director ~]$ sudo chmod 0440 /etc/sudoers.d/stack 

7. Restore the stack user home directory.

[root@director ~]$ tar -xzC / -f undercloud-backup-$DATE.tar.gz home/stack

8. Install the swift and glance base packages, and then restore their data.

[root@director ~]$ yum install -y openstack-glance openstack-swift
[root@director ~]$ tar --xattrs -xzC / -f undercloud-backup-$DATE.tar.gz srv/node var/lib/glance/images

9. Confirm the data is owned by the correct user.

[root@director ~]$ chown -R swift: /srv/node
[root@director ~]$ chown -R glance: /var/lib/glance/images

10. Restore the undercloud SSL certificates (Optional - to be done only if the setup uses SSL certificates).

[root@director ~]$ tar -xzC / -f undercloud-backup-$DATE.tar.gz etc/pki/instack-certs/undercloud.pem
[root@director ~]$ tar -xzC / -f undercloud-backup-$DATE.tar.gz etc/pki/ca-trust/source/anchors/ca.crt.pem

11. Re-run the undercloud installation as the stackuser, making sure to run it in the stack user home directory:

[root@director ~]$ su - stack
[stack@director ~]$ sudo yum install -y python-tripleoclient

12. Confirm that the hostname is correctly set in /etc/hosts.

13. Reinstall the undercloud.

[stack@director ~]$ openstack undercloud install

<snip>
#############################################################################
Undercloud install complete.

The file containing this installation's passwords is at
/home/stack/undercloud-passwords.conf.

There is also a stackrc file at /home/stack/stackrc.

These files are needed to interact with the OpenStack services, and must be
secured.

#############################################################################

Reconnect the Restored Undercloud to the Overcloud

After completing the steps above, the undercloud can be expected to automatically restore its connection to the overcloud. The nodes will continue to poll Orchestration (heat) for pending tasks, using a simple HTTP request issued every few seconds.

Validate the Completed Restore

Use these commands to perform a healthcheck of your newly restored environment.

[root@director ~]$ su - stack
Last Log in: Tue Nov 28 21:27:50 EST 2017 from 192.182.255.20 on pts/0

[stack@director ~]$ source stackrc
[stack@director ~]$ nova list
+--------------------------------------+--------------------------+--------+------------+-------------+------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------------------------+--------+------------+-------------+------------------------+
| 03f15071-21aa-4bcf-8fdd-acdbde305168 | pod2-stack-compute-0 | ACTIVE | - | Running | ctlplane=192.200.0.106 |
| 1f725ce3-948d-49e9-aed9-b99e73d82644 | pod2-stack-compute-1 | ACTIVE | - | Running | ctlplane=192.200.0.107 |
| fbc13c78-dc06-4ac9-a3c5-595ccc147adc | pod2-stack-compute-2 | ACTIVE | - | Running | ctlplane=192.200.0.119 |
| 3b94e0b1-47dc-4960-b3eb-d02ffe9ae693 | pod2-stack-compute-3 | ACTIVE | - | Running | ctlplane=192.200.0.112 |
| 5dbac94d-19b9-493e-a366-1e2e2e5e34c5 | pod2-stack-compute-4 | ACTIVE | - | Running | ctlplane=192.200.0.116 |
| b896c73f-d2c8-439c-bc02-7b0a2526dd70 | pod2-stack-controller-0 | ACTIVE | - | Running | ctlplane=192.200.0.113 |
| 2519ce67-d836-4e5f-a672-1a915df75c7c | pod2-stack-controller-1 | ACTIVE | - | Running | ctlplane=192.200.0.105 |
| e19b9625-5635-4a52-a369-44310f3e6a21 | pod2-stack-controller-2 | ACTIVE | - | Running | ctlplane=192.200.0.120 |
| 6810c884-1cb9-4321-9a07-192443920f1f | pod2-stack-osd-compute-0 | ACTIVE | - | Running | ctlplane=192.200.0.109 |
| 26d3f7b1-ba97-431f-aa6e-ba91661db45d | pod2-stack-osd-compute-1 | ACTIVE | - | Running | ctlplane=192.200.0.117 |
| 6e4a8aa9-4870-465a-a7e2-0932ff55e34b | pod2-stack-osd-compute-2 | ACTIVE | - | Running | ctlplane=192.200.0.103 |
+--------------------------------------+--------------------------+--------+------------+-------------+------------------------+
<snip>

[stack@director ~]$ ssh heat-admin@192.200.0.113
Last login: Fri Jul 6 09:02:41 2018 from 192.200.0.1 
[heat-admin@pod2-stack-controller-0 ~]$ sudo pcs status
Cluster name: tripleo_cluster
Stack: corosync
Current DC: pod2-stack-controller-2 (version 1.1.15-11.el7_3.4-e174ec8) - partition with quorum
Last updated: Tue Jul 10 10:04:15 2018Last change: Fri Jul 6 09:03:35 2018 by root via crm_attribute on pod2-stack-controller-0

3 nodes and 19 resources configured

Online: [ pod2-stack-controller-0 pod2-stack-controller-1 pod2-stack-controller-2 ]

Full list of resources:

 ip-11.120.0.49(ocf::heartbeat:IPaddr2):Started pod2-stack-controller-1
 Clone Set: haproxy-clone [haproxy]
 Started: [ pod2-stack-controller-0 pod2-stack-controller-1 pod2-stack-controller-2 ]
 Master/Slave Set: galera-master [galera]
 Masters: [ pod2-stack-controller-0 pod2-stack-controller-1 pod2-stack-controller-2 ]
 ip-192.200.0.110(ocf::heartbeat:IPaddr2):Started pod2-stack-controller-1
 ip-11.120.0.44(ocf::heartbeat:IPaddr2):Started pod2-stack-controller-2
 ip-11.118.0.49(ocf::heartbeat:IPaddr2):Started pod2-stack-controller-2
 Clone Set: rabbitmq-clone [rabbitmq]
 Started: [ pod2-stack-controller-0 pod2-stack-controller-1 pod2-stack-controller-2 ]
 ip-10.225.247.214(ocf::heartbeat:IPaddr2):Started pod2-stack-controller-1
 Master/Slave Set: redis-master [redis]
 Masters: [ pod2-stack-controller-2 ]
 Slaves: [ pod2-stack-controller-0 pod2-stack-controller-1 ]
 ip-11.119.0.49(ocf::heartbeat:IPaddr2):Started pod2-stack-controller-2
 openstack-cinder-volume(systemd:openstack-cinder-volume):Started pod2-stack-controller-1

Daemon Status:
 corosync: active/enabled
 pacemaker: active/enabled
 pcsd: active/enabled

[heat-admin@pod2-stack-controller-0 ~]$ sudo ceph status
 cluster eb2bb192-b1c9-11e6-9205-525400330666
 health HEALTH_OK
 monmap e1: 3 mons at {pod2-stack-controller-0=11.118.0.10:6789/0,pod2-stack-controller-1=11.118.0.11:6789/0,pod2-stack-controller-2=11.118.0.12:6789/0}
 election epoch 10, quorum 0,1,2 pod2-stack-controller-0,pod2-stack-controller-1,pod2-stack-controller-2
 osdmap e81: 12 osds: 12 up, 12 in
 flags sortbitwise,require_jewel_osds
 pgmap v23094282: 704 pgs, 6 pools, 809 GB data, 424 kobjects
 2418 GB used, 10974 GB / 13393 GB avail
 704 active+clean
 client io 1841 kB/s wr, 0 op/s rd, 220 op/s wr
[heat-admin@pod2-stack-controller-0 ~]$ exit
logout
Connection to 192.200.0.113 closed.

Check Identity Service (Keystone) Operation

This step validates Identity Service operations by querying for a list of users.

[stack@director ~]$ source stackerc
[stack@director~]$ openstack user list
+----------------------------------+------------------+
| ID | Name |
+----------------------------------+------------------+
| 4d93cc08ed314b87b4c6dce95c19558f | admin |
| c499bd1a65d94029804350e3bca19888 | aodh |
| d9b59136b2794ec1b9c18c4f7b95c674 | ceilometer |
| 4650cdca79b44f2b9d2426b7a9df9495 | glance |
| 4a6724a623f4463c971658462b505c8a | heat |
| e18ae9a5b7f14aefb9e93931dcec476e | ironic |
| 9831160669c44c10b04d6f78b7517a55 | ironic-inspector |
| 6a5d6f02b9a74dfb9f679eecfebb28e0 | mistral |
| 462c8ec91fb0485fbc7dfec6d4ecf480 | neutron |
| f3a66d84e39c47a0bdcd14d24bd72816 | nova |
| 1f16ea9035ab424a88d5f34681d3f893 | swift |
| ab4a1e5f62be4609b290c32dc93bfbce | zaqar |
| fac68a166f5f442284f7d06355369c53 | zaqar-websocket |
+----------------------------------+------------------+

[stack@director ~]$ source <overcloudrc>
[stack@director ~]$ openstack user list
+----------------------------------+------------+
| ID | Name |
+----------------------------------+------------+
| 7285a65e0c8c4447b10875d0e9c9da69 | admin |
| c7c78ac31ec24926b3eed79e7f5bd59e | neutron |
| 191c2a293beb48edb4d2c1160c9a0523 | heat |
| 1445a6ef2ae94b6f9365ffbc6b56a231 | gnocchi |
| fb8b006129554c6b8465310fcfc0ba8c | aodh |
| 5afcb816f93b4f20998c9edaf3b143a0 | nova |
| 37444af1295344809eca6058a782f871 | glance |
| e74d2d9e59f34ddfb4408a8beb88db7b | ceilometer |
| 9baab90178c141478ce7c0084e930f40 | cinder |
| 5a964c651c11402284eecb0e8f40ee33 | heat-cfn |
| 6d49ebae55da432580943d6080efcf90 | swift |
| 41a666fe64c64df593d73b4f467885be | core |
| ab004ba966bf4ddc92783fbb6045e48b | cpar |
| fce5b5c5dc7647ccbc2170b56a64db33 | cpardemo |
| 28ead2dab0ef486ba449b441715a3e04 | pcrf |
| 0aac4107e3d443dc93ca62c3928a1998 | pcrfdemo |
+----------------------------------+------------+

Uploading Images for Future Node Introspection

1. Validate /httpboot all these files “inspector.ipxe, agent.kernel, agent.ramdisk”, if not proceed the following steps to update in image.

[stack@director ~]$ ls /httpboot
inspector.ipxe
[stack@director ~]$ source stackrc 
[stack@director ~]$ cd images/
[stack@director images]$ openstack overcloud image upload --image-path /home/stack/images
Image "overcloud-full-vmlinuz" is up-to-date, skipping.
Image "overcloud-full-initrd" is up-to-date, skipping.
Image "overcloud-full" is up-to-date, skipping.
Image "bm-deploy-kernel" is up-to-date, skipping.
Image "bm-deploy-ramdisk" is up-to-date, skipping.
[stack@director images]$ ls /httpboot
agent.kernel agent.ramdisk inspector.ipxe
[stack@director images]$ 

Restarting Fencing

Fencing would be in stopped state after OSPD recovery, below procedure would enable fencing. 

[heat-admin@pod2-stack-controller-0 ~]$ sudo pcs property set stonith-enabled=true
[heat-admin@pod2-stack-controller-0 ~]$ sudo pcs status
[heat-admin@pod2-stack-controller-0 ~]$sudo pcs stonith show 

Related Information

• Back Up and Restore the Director Undercloud

• Fencing the Controller Nodes

• Technical Support & Documentation - Cisco Systems