Troubleshoot RCM-based UPF Upgrades (NSO-based) and Failures

Introduction

This document describes how to detect the RCM-based UPF (User Plane Function) upgrade failure due to IP pool flush issue.

Troubleshooting

1. From Active Redundancy Configuration Manager (RCM) controller run the command rcm show-statistics switchover-verbose  and look for the failed UPF switchover details if it has only start_ip_pool_flush field but end_ip_pool_flush is missing.

[RCM-active] rcm# rcm show-statistics switchover-verbose
Thu Jan 18 21:52:11.781 UTC+00:00
message :
{
"stats_verbose": [
{

"status": "Failed",
"started": "Jan 18 05:08:59.442",
"switchoverreason": "Planned Switchover",
"switchoverfailreason": "Old Active moved from PendingStandby to Active due to timeout in receiving Standby state (planned switchover)",
"source_endpoint": "192.168.100.3",
"destination_endpoint": "192.168.100.2",
"start_chkpt_flush": "Jan 18 05:09:01.783",
"end_chkpt_flush_each": {
"1": "Jan 18 05:09:05.776",
"10": "Jan 18 05:09:06.056",
"11": "Jan 18 05:09:05.909",
"12": "Jan 18 05:09:06.189",
"13": "Jan 18 05:09:05.991",
"14": "Jan 18 05:09:05.981",
"15": "Jan 18 05:09:05.531",
"16": "Jan 18 05:09:05.502",
"17": "Jan 18 05:09:05.751",
"18": "Jan 18 05:09:06.116",
"19": "Jan 18 05:09:05.834",
"2": "Jan 18 05:09:05.932",
"20": "Jan 18 05:09:05.889",
"21": "Jan 18 05:09:05.354",
"22": "Jan 18 05:09:05.780",
"3": "Jan 18 05:09:05.671",
"4": "Jan 18 05:09:05.996",
"5": "Jan 18 05:09:05.455",
"6": "Jan 18 05:09:05.839",
"7": "Jan 18 05:09:06.111",
"8": "Jan 18 05:09:06.019",
"9": "Jan 18 05:09:05.996"
},

"end_chkpt_flush": "Jan 18 05:09:06.189",
"start_cfg_push": "Jan 18 05:08:59.495",
"end_cfg_push": "Jan 18 05:09:01.774",
"start_ip_pool_flush": "Jan 18 05:09:01.783"  /* Observe under this line "end_ip_pool_flush" is missing */
},

2. Collect source and destination UPF syslogs covering switchover times +/- 30 minutes.

3. Dig into destination endpoint UPF (192.168.100.2 in output)  syslogs and around the time of UPF switchover and observe vpnmgr6(context n6) reporting several errors, related to IP pool/chunks.

    Example log events:

      - Jan 18 05:09:01 <UPF> evlogd: [local-60sec1.767] [srp 84220 error] [1/0/9589 <vpnmgr:6> vpnmgr_rcm.c:6235] [context: rcm, contextID: 6] [software internal system syslog] Failure in installing pool routes. Message Bounced.

-  evlogd: [local-60sec12.192] [vpn 5013 error] [1/0/9370 <vpnmgr:3> vpn_ip_pool.c:15699] [context: n6, contextID: 3]  [software internal system syslog] #012Prefix <IPv4 address of the IP pool> is not allocated to this UP: Closest chunk found with id 3145764 start_addr: <IPv4 address of the IP pool> end_addr: <IPv4 address of the IP pool>

Note: Only registered Cisco users can access internal Cisco information and tools.

Related Information

• Cisco Technical Support & Downloads