UPF Troubleshooting Information

This chapter covers the following topics related to monitoring and troubleshooting the UPF features:

Debug Logging


Important

The debug logging CLIs must be enabled with the help of System Administrator. Enabling debug logging CLIs can be resource intensive.

Use the following debug CLIs as required:

  • logg filter active facility sx level debug

  • logg filter active facility user-data level debug

  • logg filter active facility sessmgr level debug

  • logg filter active facility uplane level debug

  • logg filter active facility egtpc level debug

  • logg filter active facility gtpu level debug

  • logg filter active facility egtpu level debug

  • logg filter active facility gtpumgr level debug

  • logg filter active facility sxdemux level debug

  • logg filter active facility user-l3tunnel level debug

  • logg filter active facility aaamgr level debug

  • logg filter active facility vpp level debug

  • logg filter active facility dpath level debug

  • logg active pdu-verbosity 5

Monitoring CLI

Subscriber Level Message

Use the mon sub callid CLI command for subscriber level message.

Resource Tracking

Use the show task resources facility sessmgr all CLI command to track the CPU/Memory for PROCLET.

Service Status

Use the show service all CLI command to check the service status.

Sx Peer Status

Use the show sx peers CLI command to check the Sx peer status.

Monitoring Protocol

When using the monitor protocol command, enable option 49 for PFCP, and option 26 for GTP-U.

RAT Type-based Statistics

The RAT Type-based Statistics feature equip users to view data statistics segregated by RAT Type in UPF.

RAT Type-based data statistics in UPF maintains separate buckets. These buckets are created at Session Manager instance level. Bucket is assigned to a subscriber at the time of call-setup, based on RAT Type IE received in “Subscriber-Parameters”. If the IE is not received, “Unknown” RAT Type bucket is assigned to that subscriber. During the session, if UPF receives a new RAT Type for a subscriber, the bucket is changed accordingly.


Important

Data statistics are not checkpointed and lost during Session Recovery/ICSR. Only “Current-Subscriber” statistics are recalculated after recovery (during the time of call-audit).

Show Command and Output

The following CLI command displays node-level RAT statistics for UPF: show user-plane-service statistics rat { 5g-nr | all | eutran | unknown | wlan }

NOTES:

  • 5g-nr : Displays the data statistics for 5G NR subscribers.

  • all : Displays the data statistics for all RAT Type subscribers.

  • eutran : Displays the data statistics for EUTRAN subscribers.

  • unknown : Displays the data statistics for subscribers of unknown RAT type.

  • wlan : Displays the data statistics for WLAN subscribers.

Statistics

The following table provides description of each field.

Table 1. show user-plane-service statistics rat all
Field Description
Current Subscribers
5G NR Specifies the total number of current 5G NR subscribers.
EUTRAN Specifies the total number of current EUTRAN subscribers.
WLAN Specifies the total number of current WLAN subscribers.
Unknown Specifies the total number of current subscribers of unknown RAT type.
Data Statistics
5G NR Specifies the data statistics for 5G NR subscribers.
Uplink Specifies data statistics for 5G NR subscribers in uplink direction.
Total Pkts Specifies the total number of uplink packets for 5G NR subscribers.
Total Bytes Specifies the total number of uplink bytes for 5G NR subscribers.
Total Dropped Pkts Specifies the total number of uplink packets dropped for 5G NR subscribers.
Total Dropped Bytes Specifies the total number of uplink bytes dropped for 5G NR subscribers.
Downlink Specifies data statistics for 5G NR subscribers in downlink direction.
Total Pkts Specifies the total number of downlink packets for 5G NR subscribers.
Total Bytes Specifies the total number of downlink bytes for 5G NR subscribers.
Total Dropped Pkts Specifies the total number of downlink packets dropped for 5G NR subscribers.
Total Dropped Bytes Specifies the total number of downlink bytes dropped for 5G NR subscribers.
EUTRAN Specifies the data statistics for EUTRAN subscribers.
Uplink Specifies data statistics for EUTRAN subscribers in uplink direction.
Total Pkts Specifies the total number of uplink packets for EUTRAN subscribers.
Total Bytes Specifies the total number of uplink bytes for EUTRAN subscribers.
Total Dropped Pkts Specifies the total number of uplink packets dropped for EUTRAN subscribers.
Total Dropped Bytes Specifies the total number of uplink bytes dropped for EUTRAN subscribers.
Downlink Specifies data statistics for EUTRAN subscribers in downlink direction.
Total Pkts Specifies the total number of downlink packets for EUTRAN subscribers.
Total Bytes Specifies the total number of downlink bytes for EUTRAN subscribers.
Total Dropped Pkts Specifies the total number of downlink packets dropped for EUTRAN subscribers.
Total Dropped Bytes Specifies the total number of downlink bytes dropped for EUTRAN subscribers.
WLAN Specifies the data statistics for WLAN subscribers.
Uplink Specifies data statistics for WLAN subscribers in uplink direction.
Total Pkts Specifies the total number of uplink packets for WLAN subscribers.
Total Bytes Specifies the total number of uplink bytes for WLAN subscribers.
Total Dropped Pkts Specifies the total number of uplink packets dropped for WLAN subscribers.
Total Dropped Bytes Specifies the total number of uplink bytes dropped for WLAN subscribers.
Downlink Specifies data statistics for WLAN subscribers in downlink direction.
Total Pkts Specifies the total number of downlink packets for WLAN subscribers.
Total Bytes Specifies the total number of downlink bytes for WLAN subscribers.
Total Dropped Pkts Specifies the total number of downlink packets dropped for WLAN subscribers.
Total Dropped Bytes Specifies the total number of downlink bytes dropped for WLAN subscribers.
Unknown Specifies the data statistics for subscribers of unknown RAT type.
Uplink Specifies data statistics for unknown RAT type subscribers in uplink direction.
Total Pkts Specifies the total number of uplink packets for unknown RAT type subscribers.
Total Bytes Specifies the total number of uplink bytes for unknown RAT type subscribers.
Total Dropped Pkts Specifies the total number of uplink packets dropped for unknown RAT type subscribers.
Total Dropped Bytes Specifies the total number of uplink bytes dropped for unknown RAT type subscribers.
Downlink Specifies data statistics for unknown RAT type subscribers in downlink direction.
Total Pkts Specifies the total number of downlink packets for unknown RAT type subscribers.
Total Bytes Specifies the total number of downlink bytes for unknown RAT type subscribers.
Total Dropped Pkts Specifies the total number of downlink packets dropped for unknown RAT type subscribers.
Total Dropped Bytes Specifies the total number of downlink bytes dropped for unknown RAT type subscribers.

Bulk Statistics

The following bulk statistics are included in the User Plane Service schema to track RAT Type-based data statistics events.

Table 2. show bulkstats variables user-plane-service

Variable Name

Data Type

Key

Counter Type

vpnname

String

1

Info

vpnid

Int32

1

Info

servname

String

1

Info

servid

Int32

1

Info

curr-pdn-rat-eutran

Int64

0

Gauge

curr-pdn-rat-5g-nr

Int64

0

Gauge

curr-pdn-rat-wlan

Int64

0

Gauge

curr-pdn-rat-unknown

Int64

0

Gauge

uplink-total-pkts-pdn-rat-eutran

Int64

0

Counter

uplink-total-bytes-pdn-rat-eutran

Int64

0

Counter

uplink-total-pkts-dropped-pdn-rat-eutran

Int64

0

Counter

uplink-total-bytes-dropped-pdn-rat-eutran

Int64

0

Counter

downlink-total-pkts-pdn-rat-eutran

Int64

0

Counter

downlink-total-bytes-pdn-rat-eutran

Int64

0

Counter

downlink-total-pkts-dropped-pdn-rat-eutran

Int64

0

Counter

downlink-total-bytes-dropped-pdn-rat-eutran

Int64

0

Counter

uplink-total-pkts-pdn-rat-5g-nr

Int64

0

Counter

uplink-total-bytes-pdn-rat-5g-nr

Int64

0

Counter

uplink-total-pkts-dropped-pdn-rat-5g-nr

Int64

0

Counter

uplink-total-bytes-dropped-pdn-rat-5g-nr

Int64

0

Counter

downlink-total-pkts-pdn-rat-5g-nr

Int64

0

Counter

downlink-total-bytes-pdn-rat-5g-nr

Int64

0

Counter

downlink-total-pkts-dropped-pdn-rat-5g-nr

Int64

0

Counter

downlink-total-bytes-dropped-pdn-rat-5g-nr

Int64

0

Counter

uplink-total-pkts-pdn-rat-wlan

Int64

0

Counter

uplink-total-bytes-pdn-rat-wlan

Int64

0

Counter

uplink-total-pkts-dropped-pdn-rat-wlan

Int64

0

Counter

uplink-total-bytes-dropped-pdn-rat-wlan

Int64

0

Counter

downlink-total-pkts-pdn-rat-wlan

Int64

0

Counter

downlink-total-bytes-pdn-rat-wlan

Int64

0

Counter

downlink-total-pkts-dropped-pdn-rat-wlan

Int64

0

Counter
downlink-total-bytes-dropped-pdn-rat-wlan

Int64

0

Counter

uplink-total-pkts-pdn-rat-unknown

Int64

0

Counter

uplink-total-bytes-pdn-rat-unknown

Int64

0

Counter

uplink-total-pkts-dropped-pdn-rat-unknown

Int64

0

Counter

uplink-total-bytes-dropped-pdn-rat-unknown

Int64

0

Counter

downlink-total-pkts-pdn-rat-unknown

Int64

0

Counter

downlink-total-bytes-pdn-rat-unknown

Int64

0

Counter

downlink-total-pkts-dropped-pdn-rat-unknown

Int64

0

Counter

downlink-total-bytes-dropped-pdn-rat-unknown

Int64

0

Counter

Subscriber Level CLI

Use the following subscriber level CLIs as required:

  • show subscribers user-plane-only full all

  • show subscribers user-plane-only { seid seid_value | callid callid_value } pdr all

  • show subscribers user-plane-only { seid seid_value | callid callid_value } far all

  • show subscribers user-plane-only { seid seid_value | callid callid_value } qer all

  • show subscribers user-plane-only { seid seid_value | callid callid_value } urr all

  • show subscribers user-plane-only { seid seid_value | callid callid_value } bar all

  • show subscribers user-plane-only { seid seid_value | callid callid_value } pdr full all

  • show subscribers user-plane-only { seid seid_value | callid callid_value } urr full all

  • show subscribers user-plane-only { seid seid_value | callid callid_value } far full all

  • show subscribers user-plane-only { seid seid_value | callid callid_value } qer full all

  • show subscribers user-plane-only { seid seid_value | callid callid_value } bar full all

  • show subscribers user-plane-only { seid seid_value | callid callid_value } pdr id id_value

  • show subscribers user-plane-only { seid seid_value | callid callid_value } flows full

  • show subscribers user-plane-only { seid seid_value | callid callid_value } bli full all

VPP Statistics

To determine if the flows are offloaded to VPP, check for Fastpath statistics in the output of the following CLI commands:

  • show user-plane-service statistics all

  • show user-plane-service statistics analyzer name ip [ verbose ]

  • show user-plane-service statistics analyzer name ipv6 [ verbose ]

  • show user-plane-service statistics analyzer name tcp [ verbose ]

  • show user-plane-service statistics analyzer name udp [ verbose ]

  • show user-plane-service statistics analyzer name http [ verbose ]

  • show user-plane-service statistics analyzer name rtp [ verbose ]

  • show subscribers user-plane-only full callid call_id

SNMP Support

The system uses the Simple Network Management Protocol (SNMP) to send traps or events to the EMS server or an alarm server on the network. You must configure SNMP settings to communicate with those devices.

The SNMP MIB Reference describes the MIBs and SNMP traps supported by UPF and StarOS.

The following SNMP traps are available in support of their respective feature/functionality:

N4 Session/Node Level Reporting Procedure

The following traps are available to track status and conditions GTP-U path failure:

  • EGTPUPathFailure: This trap is generated when no response is received for GTP-U ECHO requests and data path failure is detected towards peer EPC Node.

  • EGTPUPathFailureClear: This trap is generated when the data path towards the peer node is available.

UP Session Recovery

The following traps are available after session recovery in the User Plane node:

  • ManagerFailure: This trap is generated when there is failure in the Software manager.

  • TaskFailed: This trap is generated when a noncritical task has failed and the appropriate recovery steps begin.

  • TaskRestart: This trap is generated when a noncritical task has restarted after an earlier failure.

  • SessMgrRecoveryComplete: This trap is generated when Session Manager recovery completes. This is typically caused by the failure of Session Manager task and successful completion of recovery.

  • ManagerRestart: This trap is generated when the identified manager task has been restarted.

Sx Association

The following traps are available to track the status of an Sx Association:

  • SxPeerAssociated: This trap is triggered when an Sx association is detected.

  • SxPeerAssociationRelease: This trap is triggered when an Sx association release is detected.

URL Blacklisting

The following SNMP trap are available in support of URL Blacklisting feature:

  • BLDBError: Specifies the blacklisting OPTBLDB file error displayed with an error code.

  • BLDBErrorClear: Specifies the blacklisting OPTBLDB file error removed.

  • BLDBUpgradeError: Specifies the blacklisting OPTBLDB file error displayed with an error code.

  • BLDBUpgradeErrorClear: Specifies the Blacklisting OPTBLDB file error removed.

Enabling SNMP Traps

Use the following configuration to enable an SNMP trap.

configure 
   snmp trap enable trap_name 
   end

For supplemental information about SNMP Support, see Management Settings chapter in the ASR 5500 System Administration Guide.

Troubleshooting UPF Features

N4 or Datapath

The following CLI commands are available for troubleshooting N4 or datapath related issues:

  • show gtpu statistics

  • show user-plane-service { all | bandwidth-policy | charging-action | edr-format | group-of-ruledefs | gtpp-group | name | pdn-instance | rulebase | ruledef | statistics | xheader-format }

    NOTES:

    • all : Displays all User Plane services.

    • bandwidth-policy : Displays information for bandwidth-policy in User Plane service.

    • charging-action : Displays information for Charging actions in User Plane service.

    • edr-format : Displays information for EDR format in user Plane service.

    • group-of-ruledefs : Displays information on Group of Ruledefs configured in User Plane service.

    • gtpp-group : Displays information for bandwidth policy in User Plane service.

    • name : Displays information for specific User Plane service name.

    • pdn-instance : Displays information for PDN instance.

    • rulebase : Displays information for rulebase in User Plane service.

    • ruledef : Displays information for ruledef in User Plane service.

    • statistics : Displays node-level statistics for User Plane.

      Additionally, you can also use: show user-plane-service statistics { all | analyzer | charging-action | fapi | rulebase | tethering-detection }

    • xheader-format : Displays information for X-Header format in User Plane service.

  • show user-plane-service content-filtering category policy-id ( all | id id_value }

    • content-filtering : Displays content filtering information.

    • category : Displays content filtering category information.

    • policy-id : Displays content filtering category Policy-ID and its definition.

    • all : Displays definitions of all content filtering category policies.

    • id id_value : Displays content filtering category definition of a particular Policy-ID. id_value is an integer ranging from 1 through 4,294,967,295.

  • show sx-service { all | name | statistics }

    NOTES:

    • all : Displays all Sx Services.

    • name : Displays information for specific Sx Service name.

    • statistics : Displays the total of collected information for specific protocol since last restart or clear command.

Content Filtering

Use the following CLI command for troubleshooting CF related issues: show user-plane-service inline-services { content-filtering | info | url-blacklisting }

NOTES:

  • content-filtering : Displays content filtering information.

  • info : Displays information of inline services.

  • url-blacklisting : Displays URL Blacklisting parameters in User Plane service.

URL Blacklisting

Use the following CLI command for troubleshooting URL Blacklisting related issues: show user-plane-service url-blacklisting database { all | debug-only | facility | url }

NOTES:

  • all : Displays all URL Blacklisting database configurations.

  • debug-only : Displays the URL Blacklisting static database debug information.

  • facility : Displays URL Blacklisting database configuration per facility.

  • url : Displays particular database information for URL Blacklisting.