Contents

Introduction

Background Information

Abbreviations

MoP Workflow

Single HDD Failure

Single HDD Failure on Compute Server

Identify VMs Hosted in Compute Node

Health Checks

Single HDD Failure on Controller Server

Single HDD Failure on OSD-Compute Server

Single HDD Failure on OSPD Server

Introduction

This document describes the steps required to replace the faulty Hard Disk Drive (HDD) in a server in an Ultra-M setup.

This procedure applies for an Openstack environment with NEWTON version where ESC does not manage CPAR and CPAR is installed directly on the Virtual Machine (VM) deployed on the Openstack.

Background Information

Ultra-M is a pre-packaged and validated virtualized mobile packet core solution designed in order to simplify the deployment of Virtual Network Functions (VNFs). OpenStack is the Virtual Infrastructure Manager (VIM) for Ultra-M and consists of these node types:

- Compute

- Object Storage Disk - Compute (OSD - Compute)

- Controller

- OpenStack Platform - Director (OSPD)

The high-level architecture of Ultra-M and the components involved are depicted in this image:

This document is intended for the Cisco personnel familiar with Cisco Ultra-M platform and it details the steps required to be carried out at OpenStack level at the time of the OSPD Server replacement.

Note: Ultra M 5.1.x release is considered in order to define the procedures in this document.

Abbreviations

VNF

Virtual Network Function

MoP

Method of Procedure

OSD

Object Storage Disks

OSPD

OpenStack Platform Director

HDD

Hard Disk Drive

SSD

Solid State Drive

VIM

Virtual Infrastructure Manager

VM

Virtual Machine

EM

Element Manager

UAS

Ultra Automation Services

UUID

Universally Unique Identifier

MoP Workflow

Single HDD Failure

1. Each Baremetal server is provisioned with two HDD drives in order to act as BOOT DISK in Raid 1 configuration. In case of single HDD failure, since there is RAID 1 level redundancy, the faulty HDD drive can be hot swapped.

2. Procedure to replace a faulty component on UCS C240 M4 server can be referred from: Replacing the Server Components.

3. In case of single HDD failure, only the faulty HDD is hot swapped and hence no BIOS upgrade procedure is required after the replacement of new disks.

4. After the replacement of the disks, you must wait for the data sync between the disks. It might take hours to complete.

5. In Openstack based (Ultra-M) solution, UCS 240M4 baremetal server can take up one of these roles: Compute, OSD-Compute, Controller and OSPD. The steps required for handling single HDD failure in each of these server roles are the same and the section here describes the health checks to be performed before the hot swap of the disk.

Single HDD Failure on Compute Server

1. If the failure of HDD drives are observed in UCS 240M4 which acts as compute node, do this health checks before you perform hot swap of the faulty disk.

2. Identify the VMs that runs on this server and verify that the status of the functions are good.

Identify VMs Hosted in Compute Node

Identify the VMs that are hosted on the compute server and verify that they are active and running.

[stack@director ~]$ nova list

| 46b4b9eb-a1a6-425d-b886-a0ba760e6114 | AAA-CPAR-testing-instance | pod2-stack-compute-4.localdomain |

Health Checks

Step 1. Run the command /opt/CSCOar/bin/arstatus at Operating System (OS) level.

[root@aaa04 ~]# /opt/CSCOar/bin/arstatus

Cisco Prime AR RADIUS server running       (pid: 24834)

Cisco Prime AR Server Agent running        (pid: 24821)

Cisco Prime AR MCD lock manager running    (pid: 24824)

Cisco Prime AR MCD server running          (pid: 24833)

Cisco Prime AR GUI running                 (pid: 24836)

SNMP Master Agent running                (pid: 24835)

[root@wscaaa04 ~]#

Step 2. Run the command /opt/CSCOar/bin/aregcmd at OS level and enter the admin credentials. Verify that the CPAR Health is 10 out of 10 and the exit CPAR CLI.

[root@aaa02 logs]# /opt/CSCOar/bin/aregcmd

Cisco Prime Access Registrar 7.3.0.1 Configuration Utility

Copyright (C) 1995-2017 by Cisco Systems, Inc.  All rights reserved.

Cluster:

User: admin

Passphrase:

Logging in to localhost

[ //localhost ]

LicenseInfo = PAR-NG-TPS 7.2(100TPS:)

PAR-ADD-TPS 7.2(2000TPS:)

PAR-RDDR-TRX 7.2()

PAR-HSS 7.2()

Radius/

Administrators/

Server 'Radius' is Running, its health is 10 out of 10

--> exit

Step 3. Run the command netstat | grep diameter and verify that all Diameter Routing Agent (DRA) connections are established.

The output mentioned here is for an environment where Diameter links are expected. If fewer links are displayed, this represents a disconnection from the DRA that needs to be analyzed.

[root@aa02 logs]# netstat | grep diameter

tcp         0            0 aaa02.aaa.epc.:77 mp1.dra01.d:diameter ESTABLISHED

tcp         0            0 aaa02.aaa.epc.:36 tsa6.dra01:diameter ESTABLISHED

tcp         0            0 aaa02.aaa.epc.:47 mp2.dra01.d:diameter ESTABLISHED

tcp         0            0 aaa02.aaa.epc.:07 tsa5.dra01:diameter ESTABLISHED

tcp         0            0 aaa02.aaa.epc.:08 np2.dra01.d:diameter ESTABLISHED

Step 4. Check that the TPS log shows requests being processed by CPAR. The values highlighted represent the TPS and those are the ones you need to pay attention to.

The value of TPS must not exceed 1500.

[root@wscaaa04 ~]# tail -f /opt/CSCOar/logs/tps-11-21-2017.csv

11-21-2017,23:57:35,263,0

11-21-2017,23:57:50,237,0

11-21-2017,23:58:05,237,0

11-21-2017,23:58:20,257,0

11-21-2017,23:58:35,254,0

11-21-2017,23:58:50,248,0

11-21-2017,23:59:05,272,0

11-21-2017,23:59:20,243,0

11-21-2017,23:59:35,244,0

11-21-2017,23:59:50,233,0

Step 5. Look for any “error” or “alarm” messages in name_radius_1_log

[root@aaa02 logs]# grep -E "error|alarm" name_radius_1_log

Step 6. In order to verify the amount of memory that the CPAR process uses, run the command:

top | grep radius

[root@sfraaa02 ~]# top | grep radius

27008 root      20   0 20.228g 2.413g  11408 S 128.3  7.7   1165:41 radius

This highlighted value must be lower than 7Gb, which is the maximum allowed at application level.

Step 7. In order to verify the disk utilization, run the command  df -h.

[root@aaa02 ~]# df -h

Filesystem                       Size  Used Avail Use% Mounted on

/dev/mapper/vg_arucsvm51-lv_root 26G   21G  4.1G  84% /

tmpfs                            1.9G  268K  1.9G   1% /dev/shm

/dev/sda1                        485M   37M  424M   8% /boot

/dev/mapper/vg_arucsvm51-lv_home 23G  4.3G   17G  21% /home

This overall value must be lower than 80%, if it's more than 80% then identify the unnecessary files and clean it up.

Step 8. Verify that there is no ‘core’ file generated.

- Core file is generated in the event of application crash when CPAR is unable to handle an exception and it's generated in these two location:

[root@aaa02 ~]# cd /cisco-ar/

[root@aaa02 ~]# cd /cisco-ar/bin

There must not be any core files located in these two location. If found, raise a Cisco TAC case in order to identify the root cause of such exception and attach the core files for debugging.

- If health checks are fine, proceed with faulty disk hot swap procedure and wait for the data sync as it takes hours to complete .

Replacing the Server Components

- Repeat the health check procedures in order to confirm that the health status of the VMs hosted on compute node are restored.

Single HDD Failure on Controller Server

- If the failure of HDD drives are observed in UCS 240M4 which acts as Controller node, do these health checks before you perform hot swap of the faulty disk.

- Check the Pacemaker status on controllers.

- Log in to one of the active controllers and check the pacemaker status. All services must be running on the available controllers and stopped on the failed controller.

[heat-admin@pod2-stack-controller-0 ~]$ sudo pcs status

Cluster name: tripleo_cluster

Stack: corosync

Current DC: pod2-stack-controller-2 (version 1.1.15-11.el7_3.4-e174ec8) - partition with quorum

Last updated: Tue Jul 10 10:04:15 2018Last change: Fri Jul 6 09:03:35 2018 by root via crm_attribute on pod2-stack-controller-0

3 nodes and 19 resources configured

Online: [ pod2-stack-controller-0 pod2-stack-controller-1 pod2-stack-controller-2 ]

Full list of resources:

ip-11.120.0.49(ocf::heartbeat:IPaddr2):Started pod2-stack-controller-1

Clone Set: haproxy-clone [haproxy]

Started: [ pod2-stack-controller-0 pod2-stack-controller-1 pod2-stack-controller-2 ]

Master/Slave Set: galera-master [galera]

Masters: [ pod2-stack-controller-0 pod2-stack-controller-1 pod2-stack-controller-2 ]

ip-192.200.0.110(ocf::heartbeat:IPaddr2):Started pod2-stack-controller-1

ip-11.120.0.44(ocf::heartbeat:IPaddr2):Started pod2-stack-controller-2

ip-11.118.0.49(ocf::heartbeat:IPaddr2):Started pod2-stack-controller-2

Clone Set: rabbitmq-clone [rabbitmq]

Started: [ pod2-stack-controller-0 pod2-stack-controller-1 pod2-stack-controller-2 ]

ip-10.225.247.214(ocf::heartbeat:IPaddr2):Started pod2-stack-controller-1

Master/Slave Set: redis-master [redis]

Masters: [ pod2-stack-controller-2 ]

Slaves: [ pod2-stack-controller-0 pod2-stack-controller-1 ]

ip-11.119.0.49(ocf::heartbeat:IPaddr2):Started pod2-stack-controller-2

openstack-cinder-volume(systemd:openstack-cinder-volume):Started pod2-stack-controller-1

Daemon Status:

corosync: active/enabled

pacemaker: active/enabled

pcsd: active/enabled

- Check MariaDB status in the active controllers.

[stack@director ~]$ nova list | grep control

| b896c73f-d2c8-439c-bc02-7b0a2526dd70 | pod2-stack-controller-0 | ACTIVE | - | Running | ctlplane=192.200.0.113 |

| 2519ce67-d836-4e5f-a672-1a915df75c7c | pod2-stack-controller-1 | ACTIVE | - | Running | ctlplane=192.200.0.105 |

| e19b9625-5635-4a52-a369-44310f3e6a21 | pod2-stack-controller-2 | ACTIVE | - | Running | ctlplane=192.200.0.120 |

[stack@director ~]$ for i in 192.200.0.102 192.200.0.110 ; do echo "*** $i ***" ; ssh heat-admin@$i "sudo mysql --exec=\"SHOW STATUS LIKE 'wsrep_local_ state_comment'\" ; sudo mysql --exec=\"SHOW STATUS LIKE 'wsrep_cluster_size'\""; done 192.200.0.110 ; do echo "*** $i ***" ; ssh heat-admin@$i "sudo mysql --exec=\"SHOW STATUS LIKE 'wsrep_local_st5 192.200.0.110 ; do echo "*** $i ***" ; ssh heat-admin@$i "sudo mysql --exec=\"SHOW STATUS LIKE 'wsrep_local_st ; do echo "*** $i ***" ; ssh heat-admin@$i "sudo mysql --exec=\"SHOW STATUS LIKE 'wsrep_local_st3 ; do echo "*** $i ***" ; ssh heat-admin@$i "sudo mysql --exec=\"SHOW STATUS LIKE 'wsrep_local_st ; do echo "*** $i ***" ; ssh heat-admin@$i "sudo mysql --exec=\"SHOW STATUS LIKE 'wsrep_local_s1 ; do echo "*** $i ***" ; ssh heat-admin@$i "sudo mysql --exec=\"SHOW STATUS LIKE 'wsrep_local_9 ; do echo "*** $i ***" ; ssh heat-admin@$i "sudo mysql --exec=\"SHOW STATUS LIKE 'wsrep_local2 ; do echo "*** $i ***" ; ssh heat-admin@$i "sudo mysql --exec=\"SHOW STATUS LIKE 'wsrep_loca. ; do echo "*** $i ***" ; ssh heat-admin@$i "sudo mysql --exec=\"SHOW STATUS LIKE 'wsrep_loc2 ; do echo "*** $i ***" ; ssh heat-admin@$i "sudo mysql --exec=\"SHOW STATUS LIKE 'wsrep_lo0 ; do echo "*** $i ***" ; ssh heat-admin@$i "sudo mysql --exec=\"SHOW STATUS LIKE 'wsrep_l0 ; do echo "*** $i ***" ; ssh heat-admin@$i "sudo mysql --exec=\"SHOW STATUS LIKE 'wsrep_. ; do echo "*** $i ***" ; ssh heat-admin@$i "sudo mysql --exec=\"SHOW STATUS LIKE 'wsrep0 ; do echo "*** $i ***" ; ssh heat-admin@$i "sudo mysql --exec=\"SHOW STATUS LIKE 'wsre. ; do echo "*** $i ***" ; ssh heat-admin@$i "sudo mysql --exec=\"SHOW STATUS LIKE 'wsr1 ; do echo "*** $i ***" ; ssh heat-admin@$i "sudo mysql --exec=\"SHOW STATUS LIKE 'ws2 ; do echo "*** $i ***" ; ssh heat-admin@$i "sudo mysql --exec=\"SHOW STATUS LIKE 'w0 ; do echo "*** $i ***" ; ssh heat-admin@$i "sudo mysql --exec=\"SHOW STATUS LIKE '

*** 192.200.0.102 ***

Variable_nameValue

wsrep_local_state_commentSynced

Variable_nameValue

wsrep_cluster_size2

*** 192.200.0.110 ***

Variable_nameValue

wsrep_local_state_commentSynced

Variable_nameValue

wsrep_cluster_size2

- Verify that these lines are present for each active controller:

wsrep_local_state_comment: Synced

wsrep_cluster_size: 2

- Check Rabbitmq status in the active controllers.

[heat-admin@pod2-stack-controller-0 ~]$ sudo rabbitmqctl cluster_status

Cluster status of node 'rabbit@pod2-stack-controller-0' ...

[{nodes,[{disc,['rabbit@pod2-stack-controller-0',

'rabbit@pod2-stack-controller-1',

'rabbit@pod2-stack-controller-2']}]},

{running_nodes,['rabbit@pod2-stack-controller-1',

'rabbit@pod2-stack-controller-2',

'rabbit@pod2-stack-controller-0']},

{cluster_name,<<"rabbit@pod2-stack-controller-1.localdomain">>},

{partitions,[]},

{alarms,[{'rabbit@pod2-stack-controller-1',[]},

{'rabbit@pod2-stack-controller-2',[]},

{'rabbit@pod2-stack-controller-0',[]}]}]

- If health checks are fine, proceed with faulty disk hot swap procedure and wait for the data sync as it takes hours to complete .

Replacing the Server Components

- Repeat the health check procedures in order to confirm that the health status on controller are restored.

Single HDD Failure on OSD-Compute Server

- If the failure of HDD drives are observed in UCS 240M4 which acts as OSD-Compute node, do the health checks before you perform hot swap of the faulty disk.

1. Identify the VMs Hosted in the OSD Compute Node

2. Identify the VMs that are hosted on the compute server

[stack@director ~]$ nova list

| 46b4b9eb-a1a6-425d-b886-a0ba760e6114 | AAA-CPAR-testing-instance | pod2-stack-compute-4.localdomain |

- CEPH processes are active on the osd-compute server.

[heat-admin@pod2-stack-osd-compute-1 ~]$ systemctl list-units *ceph*

UNIT LOAD ACTIVE SUB DESCRIPTION

var-lib-ceph-osd-ceph\x2d1.mount loaded active mounted /var/lib/ceph/osd/ceph-1

var-lib-ceph-osd-ceph\x2d10.mount loaded active mounted /var/lib/ceph/osd/ceph-10

var-lib-ceph-osd-ceph\x2d4.mount loaded active mounted /var/lib/ceph/osd/ceph-4

var-lib-ceph-osd-ceph\x2d7.mount loaded active mounted /var/lib/ceph/osd/ceph-7

ceph-osd@1.service loaded active running Ceph object storage daemon

ceph-osd@10.service loaded active running Ceph object storage daemon

ceph-osd@4.service loaded active running Ceph object storage daemon

ceph-osd@7.service loaded active running Ceph object storage daemon

system-ceph\x2ddisk.slice loaded active active system-ceph\x2ddisk.slice

system-ceph\x2dosd.slice loaded active active system-ceph\x2dosd.slice

ceph-mon.target loaded active active ceph target allowing to start/stop all ceph-mon@.service instances at once

ceph-osd.target loaded active active ceph target allowing to start/stop all ceph-osd@.service instances at once

ceph-radosgw.target loaded active active ceph target allowing to start/stop all ceph-radosgw@.service instances at once

ceph.target loaded active active ceph target allowing to start/stop all ceph*@.service instances at once

LOAD = Reflects whether the unit definition was properly loaded.

ACTIVE = The high-level unit activation state, i.e. generalization of SUB.

SUB = The low-level unit activation state, values depend on unit type.

14 loaded units listed. Pass --all to see loaded but inactive units, too.

To show all installed unit files use 'systemctl list-unit-files'.

- Verify that the mapping of OSD (HDD disk) to Journal (SSD) are good.

[heat-admin@pod2-stack-osd-compute-1 ~]$ sudo ceph-disk list

/dev/sda :

/dev/sda1 other, iso9660

/dev/sda2 other, xfs, mounted on /

/dev/sdb :

/dev/sdb1 ceph journal, for /dev/sdc1

/dev/sdb3 ceph journal, for /dev/sdd1

/dev/sdb2 ceph journal, for /dev/sde1

/dev/sdb4 ceph journal, for /dev/sdf1

/dev/sdc :

/dev/sdc1 ceph data, active, cluster ceph, osd.1, journal /dev/sdb1

/dev/sdd :

/dev/sdd1 ceph data, active, cluster ceph, osd.7, journal /dev/sdb3

/dev/sde :

/dev/sde1 ceph data, active, cluster ceph, osd.4, journal /dev/sdb2

/dev/sdf :

/dev/sdf1 ceph data, active, cluster ceph, osd.10, journal /dev/sdb4

- Verify that the ceph health and osd tree mapping are good.

[heat-admin@pod2-stack-osd-compute-1 ~]$ sudo ceph -s

cluster eb2bb192-b1c9-11e6-9205-525400330666

health HEALTH_OK

monmap e1: 3 mons at {pod2-stack-controller-0=11.118.0.10:6789/0,pod2-stack-controller-1=11.118.0.11:6789/0,pod2-stack-controller-2=11.118.0.12:6789/0}

election epoch 10, quorum 0,1,2 pod2-stack-controller-0,pod2-stack-controller-1,pod2-stack-controller-2

osdmap e81: 12 osds: 12 up, 12 in

flags sortbitwise,require_jewel_osds

pgmap v23095222: 704 pgs, 6 pools, 809 GB data, 424 kobjects

2418 GB used, 10974 GB / 13393 GB avail

704 active+clean

client io 1329 kB/s wr, 0 op/s rd, 122 op/s wr

[heat-admin@pod2-stack-osd-compute-1 ~]$ sudo ceph osd tree

ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY

-1 13.07996 root default

-2 4.35999 host pod2-stack-osd-compute-0

0 1.09000 osd.0 up 1.00000 1.00000

3 1.09000 osd.3 up 1.00000 1.00000

6 1.09000 osd.6 up 1.00000 1.00000

9 1.09000 osd.9 up 1.00000 1.00000

-3 4.35999 host pod2-stack-osd-compute-1

1 1.09000 osd.1 up 1.00000 1.00000

4 1.09000 osd.4 up 1.00000 1.00000

7 1.09000 osd.7 up 1.00000 1.00000

10 1.09000 osd.10 up 1.00000 1.00000

-4 4.35999 host pod2-stack-osd-compute-2

2 1.09000 osd.2 up 1.00000 1.00000

5 1.09000 osd.5 up 1.00000 1.00000

8 1.09000 osd.8 up 1.00000 1.00000

11 1.09000 osd.11 up 1.00000 1.00000

- If health checks are fine, proceed with faulty disk hot swap procedure and wait for the data sync as it takes hours to complete.

Replacing the Server Components

- Repeat the health check procedures in order to confirm that the health status of the VMs hosted on OSD-Compute node are restored.

Single HDD Failure on OSPD Server

- If the failure of HDD drives are observed in UCS 240M4 which acts as OSPD node, do the health checks before you perform hot swap of the faulty disk.

- Check the status of openstack stack and the node list.

[stack@director ~]$ source stackrc

[stack@director ~]$ openstack stack list --nested

[stack@director ~]$ ironic node-list

[stack@director ~]$ nova list

- Check if all the undercloud services are in loaded, active and running status from the OSP-D node.

[stack@director ~]$ systemctl list-units "openstack*" "neutron*" "openvswitch*"

UNIT LOAD ACTIVE SUB DESCRIPTION

neutron-dhcp-agent.service loaded active running OpenStack Neutron DHCP Agent

neutron-metadata-agent.service loaded active running OpenStack Neutron Metadata Agent

neutron-openvswitch-agent.service loaded active running OpenStack Neutron Open vSwitch Agent

neutron-server.service loaded active running OpenStack Neutron Server

openstack-aodh-evaluator.service loaded active running OpenStack Alarm evaluator service

openstack-aodh-listener.service loaded active running OpenStack Alarm listener service

openstack-aodh-notifier.service loaded active running OpenStack Alarm notifier service

openstack-ceilometer-central.service loaded active running OpenStack ceilometer central agent

openstack-ceilometer-collector.service loaded active running OpenStack ceilometer collection service

openstack-ceilometer-notification.service loaded active running OpenStack ceilometer notification agent

openstack-glance-api.service loaded active running OpenStack Image Service (code-named Glance) API server

openstack-glance-registry.service loaded active running OpenStack Image Service (code-named Glance) Registry server

openstack-heat-api-cfn.service loaded active running Openstack Heat CFN-compatible API Service

openstack-heat-api.service loaded active running OpenStack Heat API Service

openstack-heat-engine.service loaded active running Openstack Heat Engine Service

openstack-ironic-api.service loaded active running OpenStack Ironic API service

openstack-ironic-conductor.service loaded active running OpenStack Ironic Conductor service

openstack-ironic-inspector-dnsmasq.service loaded active running PXE boot dnsmasq service for Ironic Inspector

openstack-ironic-inspector.service loaded active running Hardware introspection service for OpenStack Ironic

openstack-mistral-api.service loaded active running Mistral API Server

openstack-mistral-engine.service loaded active running Mistral Engine Server

openstack-mistral-executor.service loaded active running Mistral Executor Server

openstack-nova-api.service loaded active running OpenStack Nova API Server

openstack-nova-cert.service loaded active running OpenStack Nova Cert Server

openstack-nova-compute.service loaded active running OpenStack Nova Compute Server

openstack-nova-conductor.service loaded active running OpenStack Nova Conductor Server

openstack-nova-scheduler.service loaded active running OpenStack Nova Scheduler Server

openstack-swift-account-reaper.service loaded active running OpenStack Object Storage (swift) - Account Reaper

openstack-swift-account.service loaded active running OpenStack Object Storage (swift) - Account Server

openstack-swift-container-updater.service loaded active running OpenStack Object Storage (swift) - Container Updater

openstack-swift-container.service loaded active running OpenStack Object Storage (swift) - Container Server

openstack-swift-object-updater.service loaded active running OpenStack Object Storage (swift) - Object Updater

openstack-swift-object.service loaded active running OpenStack Object Storage (swift) - Object Server

openstack-swift-proxy.service loaded active running OpenStack Object Storage (swift) - Proxy Server

openstack-zaqar.service loaded active running OpenStack Message Queuing Service (code-named Zaqar) Server

openstack-zaqar@1.service loaded active running OpenStack Message Queuing Service (code-named Zaqar) Server Instance 1

openvswitch.service loaded active exited Open vSwitch

LOAD = Reflects whether the unit definition was properly loaded.

ACTIVE = The high-level unit activation state, i.e. generalization of SUB.

SUB = The low-level unit activation state, values depend on unit type.

lines 1-43

lines 2-44 37 loaded units listed. Pass --all to see loaded but inactive units, too.

To show all installed unit files use 'systemctl list-unit-files'.

lines 4-46/46 (END) lines 4-46/46 (END) lines 4-46/46 (END) lines 4-46/46 (END) lines 4-46/46 (END)

- If health checks are fine, proceed with faulty disk hot swap procedure and wait for the data sync as it takes hours to complete.

Replacing the Server Components

- Repeat the health check procedures in order to confirm that the health status of the OSPD node are restored.