To test the Cisco NFVI pod and cloud infrastructure (host connectivity,
basic mraiadb, rabbit, ceph cluster check and RAID disks), you can use the
cloud-sanity tool available on the management node. To execute, enter:
# cd installer-<tagid>/tools
# ./cloud_sanity.py -–h
usage: cloud_sanity.py [-h] [--check CHECK]
[--list] [--verbose]
cloud sanity helper
optional arguments:
-h, --help show this help message and
exit
--check CHECK, -c CHECK
all - Run all sanity checks. [default
action]
control - Run controller sanity
checks.
compute - Run compute sanity checks.
cephmon - Run cephmon sanity checks.
cephosd - Run cephosd sanity checks.
management - Run Management node sanity
checks
--list, -l List all the available sanity
checks.
--verbose, -v Run the sanity in verbose
mode.
To list the available cloud-sanity checks, execute:
# ./cloud_sanity.py -l
Available sanity checks
-----------------------
-----------------------
1 - cloud-sanity : Management - Disk maintenance RAID Health
2 - cloud-sanity : Management - Disk maintenance VD Health
3 - cloud-sanity : Control - Ping All Controller Nodes
4 - cloud-sanity : Control - Ping internal VIP
5 - cloud-sanity : Control - Check Mariadb cluster size
6 - cloud-sanity : Control - Check RabbitMQ is running
7 - cloud-sanity : Control - Check RabbitMQ cluster status
8 - cloud-sanity : Control - Check Nova service list
9 - cloud-sanity : Control - Disk maintenance RAID Health
10 - cloud-sanity : Control - Disk maintenance VD Health
11 - cloud-sanity : Compute - Ping All Compute Nodes
12 - cloud-sanity : Compute - Check Nova Hypervisor list
13 - cloud-sanity : Compute - Disk maintenance RAID Health
14 - cloud-sanity : Compute - Disk maintenance VD Health
15 - cloud-sanity : CephMon - Check cephmon is running
16 - cloud-sanity : CephMon - CEPH cluster check
17 - cloud-sanity : CephMon - Check Ceph Mon status
18 - cloud-sanity : CephMon - Check Ceph Mon results
19 - cloud-sanity : CephOSD - Ping All Storage Nodes
20 - cloud-sanity : CephOSD - Check OSD result with osdinfo
21 - cloud-sanity : CephOSD - Check OSD result without osdinfo
Results for a test can be either passed, failed, or skipped. A skipped test indicates a test which couldn’t be run on this particalur PoD – i.e. a hardware RAID test is skipped on a node which doesn’t have hardware RAID.
A successful compute node check is shown below:
#./cloud_sanity.py -c compute
Executing Compute Cloud Sanity in quiet mode. This will take some time.
+---------+-----------------------------------------------------------+--------+
| Role | Task | Result |
+---------+-----------------------------------------------------------+--------+
| Compute | Compute - Ping All Compute Nodes *********************** | PASSED |
| | | |
| Compute | Compute - Check Nova Hypervisor list ******************* | PASSED |
| | | |
+---------+-----------------------------------------------------------+--------+
[PASSED] Cloud Sanity Compute Checks Passed
A failure example is shown below:
[root@MercTB1 tools]# ./cloud_sanity.py -c control
Executing Control Cloud Sanity in quiet mode. This will take some time.
+---------+-----------------------------------------------------------+--------+
| Role | Task | Result |
+---------+-----------------------------------------------------------+--------+
| Control | Control - Ping All Controller Nodes ******************** | PASSED |
| | | |
| Control | Control - Ping internal VIP **************************** | PASSED |
| | | |
| Control | Control - Check Mariadb cluster size ******************* | PASSED |
| | | |
| Control | Control - Check RabbitMQ is running ******************** | FAILED |
| | | |
+---------+-----------------------------------------------------------+--------+
[FAILED] FATAL ERROR occured when running sanity checks.
[NOTE] One or more testcase[s] skipped. Use --list to see the complete list.
To view the details of a failure, use the v option as shown below:
# ./cloud_sanity.py -c control -v
PLAY [Executes Cloud Sanity] **************************************************
GATHERING FACTS ***************************************************************
ok: [7.7.7.15]
ok: [7.7.7.11]
ok: [7.7.7.14]
TASK: [cloud-sanity | Control - Ping All Controller Nodes] ********************
changed: [7.7.7.15 -> localhost] => (item=7.7.7.15)
changed: [7.7.7.15 -> localhost] => (item=7.7.7.11)
changed: [7.7.7.15 -> localhost] => (item=7.7.7.14)
TASK: [cloud-sanity | Control - Ping internal VIP] ****************************
changed: [7.7.7.15 -> localhost]
TASK: [cloud-sanity | Control - Check Mariadb cluster size] *******************
changed: [7.7.7.11]
changed: [7.7.7.15]
changed: [7.7.7.14]
TASK: [cloud-sanity | Control - Check RabbitMQ is running] ********************
failed: [7.7.7.11] => {"changed": true, "cmd": "docker ps -a | grep rabbit | grep Up | awk '{print $NF}' | cut -f2 -d '_'", "delta": "0:00:00.021044", "end": "2016-08-18 23:45:34.838817", "failed": true, "failed_when_result": true, "rc": 0, "start": "2016-08-18 23:45:34.817773", "stdout_lines": [], "warnings": []}
changed: [7.7.7.15]
changed: [7.7.7.14]
FATAL: all hosts have already failed -- aborting
PLAY RECAP ********************************************************************
7.7.7.11 : ok=4 changed=3 unreachable=0 failed=1
7.7.7.14 : ok=5 changed=4 unreachable=0 failed=0
7.7.7.15 : ok=5 changed=4 unreachable=0 failed=0
[FAILED] FATAL ERROR occured when running sanity checks.
[NOTE] One or more testcase[s] skipped. Use --list to see the complete list.