Redeploy Procedure for Ultra-M AutoVNF

Available Languages

Download Options

PDF (55.4 KB)
View with Adobe Reader on a variety of devices
ePub (83.8 KB)
View in various apps on iPhone, iPad, Android, Sony Reader, or Windows Phone
Mobi (Kindle) (79.4 KB)
View on Kindle device or Kindle app on multiple devices

Updated:September 19, 2021

Document ID:217372

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Introduction

This document describes the steps required in order to redeploy an AutoVNF in an Ultra-M. The AutoVNF is responsible in order to bring up individual Virtual Network Function Manager (VNFM) and Virtual Network Function (VNF).

Pre-Checks

1. Log in to the OpenStack Platform Director (OSPD) and verify the status of Virtual Machine (VMs) in the health report.

[stack@labucs300-ospd ~]$ cat /var/log/cisco/ultram-health/*.report | grep -i xxx

[stack@labucs300-ospd ~]$ cat /var/log/cisco/ultram-health/ultram_health_uas.report
---------------------------------------------------------------------------------------------------------
 VNF-ID/VNFD-ID           | UAS Node | Status| Error Info, if any         
---------------------------------------------------------------------------------------------------------
10.10.10.40/LABPGW300-UAS | autovnf  | :-)   | LABPGW300-UAS:(alive)
                          |          |       |   labucs300-UAS-LABPGW300-UAS-core-UAS2-2:(alive)
                          |          |       |   labucs300-UAS-LABPGW300-UAS-core-UAS2-1:(alive)
10.10.10.40/LABPCF300-UAS | autovnf  | :-)   | LABPCF300-UAS:(alive)
                          |          |       |   labucs300-UAS-LABPCF300-UAS-core-UAS1-2:(alive)
                          |          |       |   labucs300-UAS-LABPCF300-UAS-core-UAS1-1:(alive)
10.10.10.45/LABPCF300-UGP | vnf-em   | :-)   | LABPCF300-UGP:(alive)
                          |          |       |   LABPCF300-LABPCF300-UGP-core-EM1-3:(alive)
                          |          |       |   LABPCF300-LABPCF300-UGP-core-EM1-2:(alive)
                          |          |       |   LABPCF300-LABPCF300-UGP-core-EM1-1:(alive)
10.10.10.45/LABPCF300-ESC | esc      | :-)   | LABPCF300-ESC:(alive)
                          |          |       |   LABPCF300-LABPCF300-ESC-core-ESC1-1:(alive)
                          |          |       |   LABPCF300-LABPCF300-ESC-core-ESC1-2:(alive)
10.10.10.45/LABPCF300-UGP | vnf      | :-)   | LABPCF300-UGP:(alive)
                          |          |       |   LABPCF300-LABPCF300-UGP-core-LABPCF300-CF-VDU-1:(alive)
                          |          |       |   LABPCF300-LABPCF300-UGP-core-LABPCF300-CF-VDU-0:(alive)
                          |          |       |   LABPCF300-LABPCF300-UGP-core-LABPCF300-SF-VDU-3:(alive)
                          |          |       |   LABPCF300-LABPCF300-UGP-core-LABPCF300-SF-VDU-2:(alive)
                          |          |       |   LABPCF300-LABPCF300-UGP-core-LABPCF300-SF-VDU-1:(alive)
                          |          |       |   LABPCF300-LABPCF300-UGP-core-LABPCF300-SF-VDU-0:(alive)
                          |          |       |   LABPCF300-LABPCF300-UGP-core-LABPCF300-SF-VDU-6:(alive)
                          |          |       |   LABPCF300-LABPCF300-UGP-core-LABPCF300-SF-VDU-5:(alive)
                          |          |       |   LABPCF300-LABPCF300-UGP-core-LABPCF300-SF-VDU-4:(alive)
10.10.10.48/LABPGW300-UGP | vnf-em   | :-)   | LABPGW300-UGP:(alive)
                          |          |       |   LABPGW300-LABPGW300-UGP-core-EM2-2:(alive)
                          |          |       |   LABPGW300-LABPGW300-UGP-core-EM2-3:(alive)
                          |          |       |   LABPGW300-LABPGW300-UGP-core-EM2-1:(alive)
10.10.10.48/LABPGW300-ESC | esc      | :-)   | LABPGW300-ESC:(alive)
                          |          |       |   LABPGW300-LABPGW300-ESC-core-ESC2-1:(alive)
                          |          |       |   LABPGW300-LABPGW300-ESC-core-ESC2-2:(alive)
10.10.10.48/LABPGW300-UGP | vnf      | :-)   | LABPGW300-UGP:(alive)
                          |          |       |   LABPGW300-LABPGW300-UGP-core-LABPGW300-SF-VDU-4:(alive)
                          |          |       |   LABPGW300-LABPGW300-UGP-core-LABPGW300-SF-VDU-5:(alive)
                          |          |       |   LABPGW300-LABPGW300-UGP-core-LABPGW300-SF-VDU-6:(alive)
                          |          |       |   LABPGW300-LABPGW300-UGP-core-LABPGW300-SF-VDU-0:(alive)
                          |          |       |   LABPGW300-LABPGW300-UGP-core-LABPGW300-SF-VDU-1:(alive)
                          |          |       |   LABPGW300-LABPGW300-UGP-core-LABPGW300-SF-VDU-2:(alive)
                          |          |       |   LABPGW300-LABPGW300-UGP-core-LABPGW300-SF-VDU-3:(alive)
                          |          |       |   LABPGW300-LABPGW300-UGP-core-LABPGW300-CF-VDU-0:(alive)
                          |          |       |   LABPGW300-LABPGW300-UGP-core-LABPGW300-CF-VDU-1:(alive)
---------------------------------------------------------------------------------------------------------

2. Verify the AutoVNF status.

[stack@labucs300-ospd ~]$ source *core
[stack@labucs300-ospd ~]$ nova list | grep LABPGW300-UAS-core-UAS2
| 8608fda4-b763-4753-95ff-2e07852098e3 | labucs300-UAS-LABPGW300-UAS-core-UAS2-1 | ACTIVE | - | Running | labucs300-UAS-LABPGW300-ORCH-NW=192.0.2.15; labucs300-UAS-LABPGW300-MGMT-NW=192.0.2.7 |
| 19f4496c-3907-4ea5-84c9-e5a6ef222392 | labucs300-UAS-LABPGW300-UAS-core-UAS2-2 | ACTIVE | - | Running | labucs300-UAS-LABPGW300-ORCH-NW=192.0.2.17; labucs300-UAS-LABPGW300-MGMT-NW=192.0.2.16

labucs300-UAS-LABPGW300-UAS-core-UAS2-2 -> VNFM2-UAS-VIP Primary
labucs300-UAS-LABPGW300-UAS-core-UAS2-1 -> Secondary

3. Log in to the AutoIT and check the heartbeat logs.

ubuntu@labucs300-autoit-2:~$ grep "'restarting'" /var/log/cisco/uas/heartbeat.log
2021-02-22 01:41:42,808 - 192.0.2.15: Notify Event: {'action': 'restart', 'source': 'heartbeat', 'event': 'restarting', 'ip': '192.0.2.15'}
2021-02-22 01:45:42,251 - 192.0.2.15: Notify Event: {'action': 'restart', 'source': 'heartbeat', 'event': 'restarting', 'ip': '192.0.2.15'}
2021-02-23 01:43:36,013 - 192.0.2.15: Notify Event: {'action': 'restart', 'source': 'heartbeat', 'event': 'restarting', 'ip': '192.0.2.15'}
2021-02-23 01:45:55,785 - 192.0.2.15: Notify Event: {'action': 'restart', 'source': 'heartbeat', 'event': 'restarting', 'ip': '192.0.2.15'}
2021-02-24 01:45:19,680 - 192.0.2.15: Notify Event: {'action': 'restart', 'source': 'heartbeat', 'event': 'restarting', 'ip': '192.0.2.15'}

ubuntu@labucs300-autoit-2:~$ cd /var/log/cisco/uas
ubuntu@labucs300-autoit-2:/var/log/cisco/uas$ grep "Rebooting Instance" uas_USPCHBWorker.log
2019-06-26 18:26:13,088 - Rebooting Instance: 19f4496c-3907-4ea5-84c9-e5a6ef222392
2019-06-29 03:45:12,710 - Rebooting Instance: 19f4496c-3907-4ea5-84c9-e5a6ef222392
2020-07-17 00:46:25,800 - Rebooting Instance: 19f4496c-3907-4ea5-84c9-e5a6ef222392
2020-07-18 00:47:13,347 - Rebooting Instance: 19f4496c-3907-4ea5-84c9-e5a6ef222392
2020-07-18 05:11:11,133 - Rebooting Instance: 19f4496c-3907-4ea5-84c9-e5a6ef222392
2020-07-18 5:16:07,333 - Rebooting Instance: 19f4496c-3907-4ea5-84c9-e5a6ef222392

AutoVNF Backup

AutoVNF is responsible to bring up individual VNFM and VNF. AutoDeploy sends the configuration required to instantiate both VNFM and VNF to AutoVNF and AutoVNF does this operation. In order to bring up VNFM,

AutoVNF talks directly to VIM/openstack and once VNFM is up, AutoVNF uses VNFM to bring up VNF.

AutoVNF has 1:1 redundancy and in UltraM setup, 2 AutoVNF VMs running in HA mode.

AutoVNF backup details:

Running configuration
ConfD CDB DB
AutoVNF logs (from each instance of AutoVNF)
Syslog configuration

Important: The backups must be performed before any activation/deactivation on the given POD/site and uploaded to the backup server.

1. Set ha_debug to ON on both Primary and Secondary AutoIT in /opt/uas_baseconfig.txt.

Note: The ha_debug flag is all capital. It is either ON or OFF.

ubuntu@labucs300-autoit-2:~$ vi /opt/uas_baseconfig.txt
ha: true
ha-vip: 172.16.181.7
ha-secret: f99d04acb84807c4c6c6c0eaad392b5c
orch-ips: 172.16.181.5,172.16.181.8
orch-intf: eth0
prov-ha-vip: 172.16.181.13
prov-intf: eth0
profile: AUTOIT
ha_debug: ON

ubuntu@labucs300-autoit-1:~$ vi /opt/uas_baseconfig.txt
ha: true
ha-vip: 172.16.181.7
ha-secret: f99d04acb84807c4c6c6c0eaad392b5c
orch-ips: 172.16.181.5,172.16.181.8
orch-intf: eth0
prov-ha-vip: 172.16.181.13
prov-intf: eth0
profile: AUTOIT
ha_debug: ON

2. Stop AutoIT service on Primary AutoIT with these commands:

ssh   ubuntu@ < AutoIT Floating IP>
sudo -i
service autoit stop

This is to prevent AutoIT from triggering automatic recovery of Ultra Automation Services (UAS) when the shutdown is initiated on later steps.

ubuntu@labucs300-autoit-2:~$ sudo -i
root@labucs300-autoit-2:~#
root@labucs300-autoit-2:~# service autoit status
autoit start/running, process 25001
root@labucs300-autoit-2:~# service autoit stop

3. Set ha_debug to ON on both Primary and Secondary AutoVNF (UAS) in the file /opt/uas_baseconfig.txt.

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-2:~$ vi /opt/uas_baseconfig.txt
ha: true
ha-vip: 192.0.2.8
ha-secret: d5a9fee60ddae4fe357677dcf1320e51
orch-ips: 192.0.2.15,192.0.2.17
orch-intf: eth0
profile: AUTOVNF
ha_debug: ON

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-2:~$ ssh ubuntu@192.0.2.15
ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-1:~$ cat /opt/uas_baseconfig.txt
ha: true
ha-vip: 192.0.2.8
ha-secret: d5a9fee60ddae4fe357677dcf1320e51
orch-ips: 192.0.2.15,192.0.2.17
orch-intf: eth0
profile: AUTOVNF
ha_debug: ON

4. Shutdown Secondary UAS from OSPD with commends.

 . corerc ; openstack server stop <VMName>

labucs300-UAS-LABPGW300-UAS-core-UAS2-2 -> VNFM2-UAS-VIP Primary
labucs300-UAS-LABPGW300-UAS-core-UAS2-1 -> Secondary

[stack@labucs300-ospd ~]$ nova list | grep LABPGW300-UAS-core-UAS2
| 8608fda4-b763-4753-95ff-2e07852098e3 | labucs300-UAS-LABPGW300-UAS-core-UAS2-1 | ACTIVE | - | Running | labucs300-UAS-LABPGW300-ORCH-NW=192.0.2.15; labucs300-UAS-LABPGW300-MGMT-NW=192.0.2.7 |
| 19f4496c-3907-4ea5-84c9-e5a6ef222392 | labucs300-UAS-LABPGW300-UAS-core-UAS2-2 | ACTIVE | - | Running | labucs300-UAS-LABPGW300-ORCH-NW=192.0.2.17; labucs300-UAS-LABPGW300-MGMT-NW=192.0.2.16

[stack@labucs300-ospd ~]$. corerc ; openstack server stop labucs300-UAS-LABPGW300-UAS-core-UAS2-1

5. Stop uas-confd and autovnf services on UAS with these commands:

service uas-confd stop
service autovnf stop

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-2:~$ sudo -i 
root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# service uas-confd status
uas-confd start/running, process 1305
root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# service autovnf status
autovnf start/running, process 24208
root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# service uas-confd stop
uas-confd stop/waiting
root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# service autovnf stop
autovnf stop/waiting

6. Take a backup of the UAS conf database and copy it to a backup server with these commands:

cd /opt/cisco/usp/uas/confd-latest/var/confd/
tar -cvf <pod>_<VNF>_UAS_cdb_backup.tar cdb/

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# cd /opt/cisco/usp/uas/confd-latest/var/confd/
root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# tar -cvf Autovnf_cdb_backup.tar cdb/
cdb/
cdb/O.cdb
cdb/C.cdb
cdb/aaa_init.xml
cdb/A.cdb
root@labucs300-uas-LABPGW300-uas-core-uas2-2:/opt/cisco/usp/uas/confd-latest/var/confd# ll Autovnf_cdb_backup_cdb_backup.tar 
total 1612
drwxr-xr-x 3 root root 4096 Jan 24 2017 ..
drwxr-xr-x 2 root root 4096 Jan 24 2017 log
drwxr-xr-x 8 root root 4096 Oct 11 11:30 webui
drwxr-xr-x 2 root root 4096 Oct 19 19:18 candidate
drwxr-xr-x 2 root root 4096 Oct 23 13:00 rollback
drwxr-xr-x 2 root root 4096 Oct 28 17:00 cdb
drwxr-xr-x 3 root root 4096 Oct 28 17:00 state
drwxr-xr-x 8 root root 4096 Oct 31 18:00 .
-rw-r--r-- 1 root root 1617920 Oct 31 18:00 Autovnf_cdb_backup.tar

7. Restart uas-confd and autovnf services on UAS with these commands:

service uas-confd start
service autovnf start

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# service uas-confd start
uas-confd start/running, process 13852
root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# service autovnf start
autovnf start/running, process 13853

8. Restart Secondary UAS from OSPD with these commands:

. corerc ; openstack server start <VMName>

[stack@labucs300-ospd ~]$. corerc ; openstack server start labucs300-UAS-LABPGW300-UAS-core-UAS2-1

9. Verify both Primary and Secondary UAS show up as alive in show uas command.

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-2:~$ sudo -i
root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# confd_cli -u admin -C
Welcome to the ConfD CLI
admin connected from 127.0.0.1 using console on labucs300-uas-LABPGW300-uas-core-uas2-2
labucs300-uas-LABPGW300-uas-core-uas2-2#show uas 
uas version 6.2.0
uas state active
uas external-connection-point 192.0.2.8
INSTANCE IP STATE ROLE 
-----------------------------------
0.0.0.0 error CONFD-Secondary
192.0.2.15 alive CONFD-Secondary
192.0.2.17 alive CONFD-Primary

10. Change ha_debug to OFF on both Primary and Secondary AutoVNF (UAS) in the file /opt/uas_baseconfig.txt.

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-2:~$ vi /opt/uas_baseconfig.txt
ha: true
ha-vip: 192.0.2.8
ha-secret: d5a9fee60ddae4fe357677dcf1320e51
orch-ips: 192.0.2.15,192.0.2.17
orch-intf: eth0
profile: AUTOVNF
ha_debug: OFF
ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-2:~$ ssh ubuntu@192.0.2.15
ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-1:~$ cat /opt/uas_baseconfig.txt
ha: true
ha-vip: 192.0.2.8
ha-secret: d5a9fee60ddae4fe357677dcf1320e51
orch-ips: 192.0.2.15,192.0.2.17
orch-intf: eth0
profile: AUTOVNF
ha_debug: OFF

11. Restart autoit service on Primary AutoIT with service autoit start command.

ubuntu@labucs300-autoit-2:~$ sudo -i
root@labucs300-autoit-2:~# service autoit start

12. Verify that both Primary and Secondary AutoIT show up as alive with show uas command.

ubuntu@labucs300-autoit-2:~$ sudo -i
root@labucs300-autoit-2:~# confd_cli -u admin -C 
Welcome to the ConfD CLI
admin connected from 127.0.0.1 using console on labucs300-autoit-2
labucs300-autoit-2#show uas
uas version 6.2.0
uas state active
uas external-connection-point 172.16.181.7
INSTANCE IP STATE ROLE 
-----------------------------------
172.16.181.5 alive CONFD-Secondary 
172.16.181.8 alive CONFD-Primary

13. Set ha_debug to OFF on both Primary and Secondary AutoIT in the file /opt/uas_baseconfig.txt.

ubuntu@labucs300-autoit-2:~$ vi /opt/uas_baseconfig.txt
ha: true
ha-vip: 172.16.181.7
ha-secret: f99d04acb84807c4c6c6c0eaad392b5c
orch-ips: 172.16.181.5,172.16.181.8
orch-intf: eth0
prov-ha-vip: 172.16.181.13
prov-intf: eth0
profile: AUTOIT
ha_debug: OFF

ubuntu@labucs300-autoit-1:~$ vi /opt/uas_baseconfig.txt
ha: true
ha-vip: 172.16.181.7
ha-secret: f99d04acb84807c4c6c6c0eaad392b5c
orch-ips: 172.16.181.5,172.16.181.8
orch-intf: eth0
prov-ha-vip: 172.16.181.13
prov-intf: eth0
profile: AUTOIT
ha_debug: OFF

14. Collect logs from the UAS and transfer them to a backup server.

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# cd /opt/cisco/usp/uas/confd-latest/var/confd/
root@labucs300-uas-LABPGW300-uas-core-uas2-2:/opt/cisco/usp/uas/confd-latest/var/confd# cd /opt/cisco/usp/uas/scripts/
root@labucs300-uas-LABPGW300-uas-core-uas2-2:/opt/cisco/usp/uas/scripts# sudo ./collect-uas-logs.sh 
Dumping output for show transaction in file /tmp/uas-logs/transactions.txt
Dumping output for show log in file /tmp/uas-logs/transactions.txt
Dumping output for show running-config in file /tmp/uas-logs/confd_output.txt
Dumping output for show uas in file /tmp/uas-logs/confd_output.txt
Dumping output for show usp in file /tmp/uas-logs/confd_output.txt
..........

15. Log in to the secondary AutoVNF and repeat previous step in order to collect the logs and transfer them to the backup server.

16. Backup the syslog configuration on the Primary and Secondary AutoVNF VMs and transfer them to the backup server. The files are located in these directories:

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# ls /etc/rsyslog.d/00-autovnf.conf 
/etc/rsyslog.d/00-autovnf.conf
root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# ls /etc/rsyslog.conf
/etc/rsyslog.conf

17. Enable autoit service with service autoit start command on Primary AutoIT.

ubuntu@labucs300-autoit-2:~$ sudo -i
root@labucs300-autoit-2:~#
root@labucs300-autoit-2:~# service autoit start
autoit start/running, process 25001

18. Set the ha_debug flag mode to OFF in the /opt/uas_baseconfig.txt on Primary AutoVNF and AutoIT.

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-2:~$ vi /opt/uas_baseconfig.txt
ha: true
ha-vip: 192.0.2.8
ha-secret: d5a9fee60ddae4fe357677dcf1320e51
orch-ips: 192.0.2.15,192.0.2.17
orch-intf: eth0
profile: AUTOVNF
ha_debug: OFF

ubuntu@labucs300-autoit-2:~$ vi /opt/uas_baseconfig.txt
ha: true
ha-vip: 172.16.181.7
ha-secret: f99d04acb84807c4c6c6c0eaad392b5c
orch-ips: 172.16.181.5,172.16.181.8
orch-intf: eth0
prov-ha-vip: 172.16.181.13
prov-intf: eth0
profile: AUTOIT
ha_debug: OFF

19. Confirm uas-confd and autovnf services running on UAS.

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# service uas-confd status
uas-confd start/running, process 1305
root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# service autovnf status
autovnf start/running, process 24208

AutoVFN Redeploy

1. Log into AutoDeploy and note the UAS instance.

ubuntu@labucs300-autodeploy-2:~$ sudo su 
root@labucs300-autodeploy-2:/home/ubuntu# confd_cli -u admin -C
Welcome to the ConfD CLI
admin connected from 127.0.0.1 using console on labucs300-autodeploy-2
labucs300-autodeploy-2#show nsr
 nsr         LABSGW300-instance
 nsd         LABSGW300
 vnfr        [ LABPCF300-LABPCF300-ESC LABPCF300-LABPCF300-UGP ]
 vnf-package [ usp_6_2_b8 ]
 vim-artifact vim_art_rack
nsr LABPGW300-instance
 nsd         LABPGW300
 vnfr        [ LABPGW300-LABPGW300-ESC LABPGW300-LABPGW300-UGP ]
 vnf-package [ usp_6_2_b8 ]
 vim-artifact vim_art_rack
nsr labucs300-UAS-instance
 nsd         labucs300-UAS
 vnfr        [ labucs300-UAS-LABPCF300-UAS labucs300-UAS-LABPGW300-UAS ]
 vnf-package [ usp_6_2_b8 ]
 vim-artifact vim_art_rack

2. Deactivate AutoVNF from AutoDeploy with deactivate nsd-id <nsd-id> vnfd <vnfd-id> command.

ubuntu@labucs300-autodeploy-2:~$ /opt/cisco/usp/uas/confd-6.3.1/bin/confd_cli -u admin -C
Welcome to the ConfD CLI
admin connected from 10.10.10.10 using ssh on labucs300-autodeploy-2
labucs300-autodeploy-2#nsd:deactivate nsd-id labucs300-UAS vnfd [LABPGW300-UAS]
transaction-id 1560431372-357328

3. Confirm the status of the transaction.

labucs300-autodeploy-2#show transaction 
                                                           DEPLOYMENT                                                 STATUS 
TX ID                               TX TYPE                ID            TIMESTAMP                        STATUS      DETAIL 
------------------------------------------------------------------------------------------------------------------------------
1560431372-357328                   activate-ns-deployment labucs300-UAS 2019-06-13T13:09:32.357355-00:00 in-progress - 
1560431372-357328/1560431373-102024 activate-ns-deployment labucs300-UAS 2019-06-13T13:09:33.102041-00:00 in-progress -

4. Check logs of the transaction. In this case labucs300-UAS transactions: 1560431372-357328 and 1560431372-357328/1560431373-102024.

labucs300-autodeploy-2#show log 1560431372-357328 | display xml
<config xmlns="http://tail-f.com/ns/config/1.0">
<log xmlns="http://www.cisco.com/usp/nfv/usp-transaction">
<tx-id>1560431372-357328</tx-id>
<log>
2019-06-13 13:09:33,367 - Send Deployment notification for: labucs300-UAS-instance
2019-06-13 13:09:33,375 - Deployment activate-ns-deployment: labucs300-UAS started
2019-06-13 13:09:33,378 - Adding NSR: labucs300-UAS-instance
2019-06-13 13:09:33,385 - Start pipeline of 1 tasks
2019-06-13 13:09:33,390 - Scheduling Task: labucs300-UAS
2019-06-13 13:09:33,400 - Waiting for all workers to finish the transactions
2019-06-13 13:15:00,006 - Deployment activate-ns-deployment: labucs300-UAS succeeded
2019-06-13 13:15:00,020 - Send Deployment notification for: labucs300-UAS-instance
2019-06-13 13:09:33,437 - Send Deployment notification for: labucs300-UAS-instance-deploy
2019-06-13 13:09:33,441 - Deployment activate-ns-deployment: labucs300-UAS started
……………

labucs300-autodeploy-2#show log 1560431372-357328/1560431373-102024 | display xml
<config xmlns="http://tail-f.com/ns/config/1.0">
<log xmlns="http://www.cisco.com/usp/nfv/usp-transaction">
<tx-id>1560431372-357328/1560431373-102024</tx-id>
<log>
2019-06-13 13:09:33,437 - Send Deployment notification for: labucs300-UAS-instance-deploy
2019-06-13 13:09:33,441 - Deployment activate-ns-deployment: labucs300-UAS started
2019-06-13 13:09:33,446 - Adding NSR: labucs300-UAS-instance, VNFR: labucs300-UAS-LABPCF300-UAS, vlrs: None
2019-06-13 13:09:33,453 - Adding NSR: labucs300-UAS-instance, VNFR: labucs300-UAS-LABPGW300-UAS, vlrs: None
2019-06-13 13:09:33,463 - VNF deployment pre-check success(all-not-present)
2019-06-13 13:09:33,472 - VNF-Package deployment pre-check success(all-not-present)
2019-06-13 13:09:33,481 - VIM-Artifact deployment pre-check success
2019-06-13 13:09:33,487 - Skipping VIM-Orch pre-deployment, since VIM-Orch is not defined
2019-06-13 13:09:33,496 - Skipping VIM pre-deployment, since VIM is not defined
2019-06-13 13:09:33,499 - NS pre-check success
2019-06-13 13:09:33,503 - Copying '/home/ubuntu/usp-6_2_b8.iso' to '/var/cisco/isos/labucs300-UAS_usp_6_2_b8'
2019-06-13 13:09:53,359 - Updated path to URL 'http://172.16.181.14:5000/isos/labucs300-UAS_usp_6_2_b8'

5. Wait until the transaction completed and confirm the state.

labucs300-autodeploy-2#show transaction 
                                                           DEPLOYMENT                                            STATUS 
TX ID                               TX TYPE                ID            TIMESTAMP                        STATUS DETAIL 
------------------------------------------------------------------------------------------------------------------------------
1560431372-357328                   activate-ns-deployment labucs300-UAS 2019-06-13T13:09:32.357355-00:00 success - 
1560431372-357328/1560431373-102024 activate-ns-deployment labucs300-UAS 2019-06-13T13:09:33.102041-00:00 success -

6. Activate AutoVNF VNFD from AutoDeploy with activate nsd-id <nsd-id> vnfd <vnfd-id> command.

ubuntu@labucs300-autodeploy-2:~$ /opt/cisco/usp/uas/confd-6.3.1/bin/confd_cli -u admin -C
Welcome to the ConfD CLI
admin connected from 10.253.110.47 using ssh on labucs300-autodeploy-2
labucs300-autodeploy-2#nsd:activate nsd-id labucs300-UAS vnfd [LABPGW300-UAS]
transaction-id 1560431371-357330

7. Check status of the transaction and collect logs from the transaction with these commands:

show transaction
show log <transaction-id> | display xml
show log <transaction-id> | display xml

8. Wait till the transactions are completed. The show transaction command shows the state of transactions.

Backup Restoration

1. On Primary AutoIT set the ha_debug flag mode to ON in /opt/uas_baseconfig.txt.

ubuntu@labucs300-autoit-2:~$ vi /opt/uas_baseconfig.txt
ha: true
ha-vip: 172.16.181.7
ha-secret: f99d04acb84807c4c6c6c0eaad392b5c
orch-ips: 172.16.181.5,172.16.181.8
orch-intf: eth0
prov-ha-vip: 172.16.181.13
prov-intf: eth0
profile: AUTOIT
ha_debug: ON

2. On Primary AutoIT stop service autoit. This step is to prevent AutoIT to automatically recover UAS.

ubuntu@labucs300-autoit-2:~$ sudo -i
root@labucs300-autoit-2:~#
root@labucs300-autoit-2:~# service autoit status
autoit start/running, process 25001
root@labucs300-autoit-2:~# service autoit stop

3. On Primary UAS set the ha_debug flag mode to ON in /opt/uas_baseconfig.txt.

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-2:~$ vi /opt/uas_baseconfig.txt
ha: true
ha-vip: 192.0.2.8
ha-secret: d5a9fee60ddae4fe357677dcf1320e51
orch-ips: 192.0.2.15,192.0.2.17
orch-intf: eth0
profile: AUTOVNF
ha_debug: ON

4. On Secondary UAS set the ha_debug flag mode to ON in /opt/uas_baseconfig.txt.

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-1:~$ cat /opt/uas_baseconfig.txt
ha: true
ha-vip: 192.0.2.8
ha-secret: d5a9fee60ddae4fe357677dcf1320e51
orch-ips: 192.0.2.15,192.0.2.17
orch-intf: eth0
profile: AUTOVNF
ha_debug: ON

5. On OSPD stop Secondary UAS server with the OpenStack command.

labucs300-UAS-LABPGW300-UAS-core-UAS2-2 -> VNFM2-UAS-VIP Primary
labucs300-UAS-LABPGW300-UAS-core-UAS2-1 -> Secondary

[stack@labucs300-ospd ~]$. corerc ; openstack server stop labucs300-UAS-LABPGW300-UAS-core-UAS2-1

6. On Primary UAS stop uas-confd service.

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-2:~$ sudo -i 
root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# service uas-confd status
uas-confd start/running, process 1305
root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# service uas-confd stop
uas-confd stop/waiting

7. On Primary UAS, copy the backuped archive of CDB to directory /opt/cisco/usp/uas/confd-latest/var/confd/.

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# cp Autovnf_cdb_backup.tar to /opt/cisco/usp/uas/confd-latest/var/confd/

8. On Primary UAS delete files under CBD directory.

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# cd /opt/cisco/usp/uas/confd-latest/var/confd/ ; rm cdb/*

9. On Primary UAS extract files from the CDB backup file.

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# cd /opt/cisco/usp/uas/confd-latest/var/confd/ ; tar -xvf <archive_backup_tar_file>

10. On OSPD reboot Primary UAS with OpenStack commands.

[stack@labucs300-ospd ~]$ source *core
[stack@labucs300-ospd ~]$ nova list | grep LABPGW300-UAS-core-UAS2
| 8608fda4-b763-4753-95ff-2e07852098e3 | labucs300-UAS-LABPGW300-UAS-core-UAS2-1 | ACTIVE | - | Running | labucs300-UAS-LABPGW300-ORCH-NW=192.0.2.15; labucs300-UAS-LABPGW300-MGMT-NW=192.0.2.7 |
| 19f4496c-3907-4ea5-84c9-e5a6ef222392 | labucs300-UAS-LABPGW300-UAS-core-UAS2-2 | ACTIVE | - | Running | labucs300-UAS-LABPGW300-ORCH-NW=192.0.2.17; labucs300-UAS-LABPGW300-MGMT-NW=192.0.2.16

labucs300-UAS-LABPGW300-UAS-core-UAS2-2 -> VNFM2-UAS-VIP Primary
labucs300-UAS-LABPGW300-UAS-core-UAS2-1 -> Secondary

[stack@labucs300-ospd ~]$ nova reboot --hard 19f4496c-3907-4ea5-84c9-e5a6ef222392
Request to reboot server <Server: auto-testautovnf1-uas-2> has been accepted.

11. Wait till the Primary UAS comes up. Check the status os UAS on Primary UAS after the reboot. The Primary state is alive while secondary one shows state unknown.

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-2:~$ sudo -i
root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# confd_cli -u admin -C
Welcome to the ConfD CLI
admin connected from 127.0.0.1 using console on labucs300-uas-LABPGW300-uas-core-uas2-2
labucs300-uas-LABPGW300-uas-core-uas2-2#show uas 
uas version 6.2.0
uas state active
uas external-connection-point 192.0.2.8
INSTANCE IP STATE ROLE 
-----------------------------------
192.0.2.15 unknown CONFD-Secondary 
192.0.2.17 alive CONFD-Primary

12. On OSPD, start Secondary UAS with the OpenStack command.

[stack@labucs300-ospd ~]$. corerc ; openstack server start labucs300-UAS-LABPGW300-UAS-core-UAS2-1

13. On OSPD, verify the states of the Primary and Secondary UAS are Active.

[stack@labucs300-ospd ~]$ openstack server list | grep labucs300-UAS-LABPGW300
| 19f4496c-3907-4ea5-84c9-e5a6ef222392 | labucs300-UAS-LABPGW300-UAS-core-UAS2-2 | ACTIVE | labucs300-UAS-LABPGW300-ORCH-NW=192.0.2.17; labucs300-UAS-LABPGW300-MGMT-NW=192.0.2.16 | labucs300-UAS-usp_6_2_b8-core-uas |
| 8608fda4-b763-4753-95ff-2e07852098e3 | labucs300-UAS-LABPGW300-UAS-core-UAS2-1 | ACTIVE | labucs300-UAS-LABPGW300-ORCH-NW=192.0.2.15; labucs300-UAS-LABPGW300-MGMT-NW=192.0.2.7 | labucs300-UAS-usp_6_2_b8-core-uas |

14. On Primary UAS verify the states of the Primary and Secondary UAS are alive.

labucs300-uas-LABPGW300-uas-core-uas2-2#show uas 
uas version 6.2.0
uas state active
uas external-connection-point 192.0.2.8
INSTANCE IP STATE ROLE 
-----------------------------------
192.0.2.15 alive CONFD-Secondary
192.0.2.17 alive CONFD-Primary

15. On Primary AutoIT start autoit service.

ubuntu@labucs300-autoit-2:~$ sudo -i
root@labucs300-autoit-2:~# service autoit start

16. Verify that the Secure Shell (SSH) session stays up for a couple of minutes to Primary and Secondary UAS.

17. On Primary UAS set the ha_debug flag mode to OFF in /opt/uas_baseconfig.txt.

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-2:~$ vi /opt/uas_baseconfig.txt
ha: true
ha-vip: 192.0.2.8
ha-secret: d5a9fee60ddae4fe357677dcf1320e51
orch-ips: 192.0.2.15,192.0.2.17
orch-intf: eth0
profile: AUTOVNF
ha_debug: OFF

18. On Secondary UAS set the ha_debug flag mode to OFF in /opt/uas_baseconfig.txt.

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-1:~$ cat /opt/uas_baseconfig.txt
ha: true
ha-vip: 192.0.2.8
ha-secret: d5a9fee60ddae4fe357677dcf1320e51
orch-ips: 192.0.2.15,192.0.2.17
orch-intf: eth0
profile: AUTOVNF
ha_debug: OFF

19. On Primary AutoIT set the ha_debug flag mode to OFF in /opt/uas_baseconfig.txt.

 ubuntu@labucs300-autoit-2:~$ vi /opt/uas_baseconfig.txt
ha: true
ha-vip: 172.16.181.7
ha-secret: f99d04acb84807c4c6c6c0eaad392b5c
orch-ips: 172.16.181.5,172.16.181.8
orch-intf: eth0
prov-ha-vip: 172.16.181.13
prov-intf: eth0
profile: AUTOIT
ha_debug: OFF

20. On AutoVNF check the 00-autovnf.conf and rsyslog.conf files and restore them from previous backup.

ubuntu@autoit-tb1-autovnf1-core-avf-1:~#sudo su
root@autoit-tb1-autovnf1-core-avf-1:~#ls /etc/rsyslog.d/00-autovnf.conf
00-autovnf.conf

root@autoit-tb1-autovnf1-core-avf-1:~#/home/ubuntu#ls /etc/rsyslog.conf
rsyslog.conf

Post-Checks

On OSPD verify that both AutoVNFs are active and check Ultra-M health check report.

[stack@labucs300-ospd ~]$ cat /var/log/cisco/ultram-health/*.report | grep -i xxx

[stack@labucs300-ospd ~]$ cat /var/log/cisco/ultram-health/ultram_health_uas.report

Revision History

Revision	Publish Date	Comments
1.0	19-Sep-2021	Initial Release

Contributed by Cisco Engineers

Dennis Lanov
Cisco TAC Engineer

Redeploy Procedure for Ultra-M AutoVNF

Available Languages

Download Options

Bias-Free Language

Contents

Introduction

Pre-Checks

AutoVNF Backup

AutoVFN Redeploy

Backup Restoration

Post-Checks

Revision History

Contributed by Cisco Engineers

Was this Document Helpful?

Contact Cisco

This Document Applies to These Products