UPF Troubleshooting Information
Debug Logging
Monitoring CLI
Monitoring Protocol
RAT Type-based Statistics
Subscriber Level CLI
- show subscribers user-plane-only full msid msid_value
VPP Statistics
SNMP Support
Troubleshooting UPF Features

UPF Troubleshooting Information

This chapter covers the following topics related to monitoring and troubleshooting the UPF features:

Debug Logging

Important

The debug logging CLIs must be enabled with the help of System Administrator. Enabling debug logging CLIs can be resource intensive.

Use the following debug CLIs as required:

logg filter active facility sx level debug
logg filter active facility user-data level debug
logg filter active facility sessmgr level debug
logg filter active facility uplane level debug
logg filter active facility egtpc level debug
logg filter active facility gtpu level debug
logg filter active facility egtpu level debug
logg filter active facility gtpumgr level debug
logg filter active facility sxdemux level debug
logg filter active facility user-l3tunnel level debug
logg filter active facility aaamgr level debug
logg filter active facility vpp level debug
logg filter active facility dpath level debug
logg active pdu-verbosity 5
logg syslog ip_address facility facilities event-verbosity { min | concise | full }

Monitoring CLI

Subscriber Level Message

Use the mon sub callid CLI command for subscriber level message.

Resource Tracking

Use the show task resources facility sessmgr all CLI command to track the CPU/Memory for PROCLET.

Service Status

Use the show service all CLI command to check the service status.

Sx Peer Status

Use the show sx peers CLI command to check the Sx peer status.

Monitoring Protocol

When using the monitor protocol command, enable option 49 for PFCP, and option 26 for GTP-U.

RAT Type-based Statistics

The RAT Type-based Statistics feature equip users to view data statistics segregated by RAT Type in UPF.

RAT Type-based data statistics in UPF maintains separate buckets. These buckets are created at Session Manager instance level. Bucket is assigned to a subscriber at the time of call-setup, based on RAT Type IE received in “Subscriber-Parameters”. If the IE is not received, “Unknown” RAT Type bucket is assigned to that subscriber. During the session, if UPF receives a new RAT Type for a subscriber, the bucket is changed accordingly.

Important

Data statistics are not checkpointed and lost during Session Recovery/ICSR. Only “Current-Subscriber” statistics are recalculated after recovery (during the time of call-audit).

Show Command and Output

The following CLI command displays node-level RAT statistics for UPF: show user-plane-service statistics rat { 5g-nr | all | eutran | unknown | wlan }

NOTES:

5g-nr : Displays the data statistics for 5G NR subscribers.
all : Displays the data statistics for all RAT Type subscribers.
eutran : Displays the data statistics for EUTRAN subscribers.
unknown : Displays the data statistics for subscribers of unknown RAT type.
wlan : Displays the data statistics for WLAN subscribers.

Statistics

The following table provides description of each field.

Table 1. show user-plane-service statistics rat all
Field	Description
Current Subscribers
5G NR	Specifies the total number of current 5G NR subscribers.
EUTRAN	Specifies the total number of current EUTRAN subscribers.
WLAN	Specifies the total number of current WLAN subscribers.
Unknown	Specifies the total number of current subscribers of unknown RAT type.
Data Statistics
5G NR	Specifies the data statistics for 5G NR subscribers.
Uplink	Specifies data statistics for 5G NR subscribers in uplink direction.
Total Pkts	Specifies the total number of uplink packets for 5G NR subscribers.
Total Bytes	Specifies the total number of uplink bytes for 5G NR subscribers.
Total Dropped Pkts	Specifies the total number of uplink packets dropped for 5G NR subscribers.
Total Dropped Bytes	Specifies the total number of uplink bytes dropped for 5G NR subscribers.
Downlink	Specifies data statistics for 5G NR subscribers in downlink direction.
Total Pkts	Specifies the total number of downlink packets for 5G NR subscribers.
Total Bytes	Specifies the total number of downlink bytes for 5G NR subscribers.
Total Dropped Pkts	Specifies the total number of downlink packets dropped for 5G NR subscribers.
Total Dropped Bytes	Specifies the total number of downlink bytes dropped for 5G NR subscribers.

EUTRAN	Specifies the data statistics for EUTRAN subscribers.
Uplink	Specifies data statistics for EUTRAN subscribers in uplink direction.
Total Pkts	Specifies the total number of uplink packets for EUTRAN subscribers.
Total Bytes	Specifies the total number of uplink bytes for EUTRAN subscribers.
Total Dropped Pkts	Specifies the total number of uplink packets dropped for EUTRAN subscribers.
Total Dropped Bytes	Specifies the total number of uplink bytes dropped for EUTRAN subscribers.
Downlink	Specifies data statistics for EUTRAN subscribers in downlink direction.
Total Pkts	Specifies the total number of downlink packets for EUTRAN subscribers.
Total Bytes	Specifies the total number of downlink bytes for EUTRAN subscribers.
Total Dropped Pkts	Specifies the total number of downlink packets dropped for EUTRAN subscribers.
Total Dropped Bytes	Specifies the total number of downlink bytes dropped for EUTRAN subscribers.

WLAN	Specifies the data statistics for WLAN subscribers.
Uplink	Specifies data statistics for WLAN subscribers in uplink direction.
Total Pkts	Specifies the total number of uplink packets for WLAN subscribers.
Total Bytes	Specifies the total number of uplink bytes for WLAN subscribers.
Total Dropped Pkts	Specifies the total number of uplink packets dropped for WLAN subscribers.
Total Dropped Bytes	Specifies the total number of uplink bytes dropped for WLAN subscribers.
Downlink	Specifies data statistics for WLAN subscribers in downlink direction.
Total Pkts	Specifies the total number of downlink packets for WLAN subscribers.
Total Bytes	Specifies the total number of downlink bytes for WLAN subscribers.
Total Dropped Pkts	Specifies the total number of downlink packets dropped for WLAN subscribers.
Total Dropped Bytes	Specifies the total number of downlink bytes dropped for WLAN subscribers.

Unknown	Specifies the data statistics for subscribers of unknown RAT type.
Uplink	Specifies data statistics for unknown RAT type subscribers in uplink direction.
Total Pkts	Specifies the total number of uplink packets for unknown RAT type subscribers.
Total Bytes	Specifies the total number of uplink bytes for unknown RAT type subscribers.
Total Dropped Pkts	Specifies the total number of uplink packets dropped for unknown RAT type subscribers.
Total Dropped Bytes	Specifies the total number of uplink bytes dropped for unknown RAT type subscribers.
Downlink	Specifies data statistics for unknown RAT type subscribers in downlink direction.
Total Pkts	Specifies the total number of downlink packets for unknown RAT type subscribers.
Total Bytes	Specifies the total number of downlink bytes for unknown RAT type subscribers.
Total Dropped Pkts	Specifies the total number of downlink packets dropped for unknown RAT type subscribers.
Total Dropped Bytes	Specifies the total number of downlink bytes dropped for unknown RAT type subscribers.

Bulk Statistics

The following bulk statistics are included in the User Plane Service schema to track RAT Type-based data statistics events.

Table 2. show bulkstats variables user-plane-service
Variable Name	Data Type	Key	Counter Type
vpnname	String	1	Info
vpnid	Int32	1	Info
servname	String	1	Info
servid	Int32	1	Info
curr-pdn-rat-eutran	Int64	0	Gauge
curr-pdn-rat-5g-nr	Int64	0	Gauge
curr-pdn-rat-wlan	Int64	0	Gauge
curr-pdn-rat-unknown	Int64	0	Gauge
uplink-total-pkts-pdn-rat-eutran	Int64	0	Counter
uplink-total-bytes-pdn-rat-eutran	Int64	0	Counter
uplink-total-pkts-dropped-pdn-rat-eutran	Int64	0	Counter
uplink-total-bytes-dropped-pdn-rat-eutran	Int64	0	Counter
downlink-total-pkts-pdn-rat-eutran	Int64	0	Counter
downlink-total-bytes-pdn-rat-eutran	Int64	0	Counter
downlink-total-pkts-dropped-pdn-rat-eutran	Int64	0	Counter
downlink-total-bytes-dropped-pdn-rat-eutran	Int64	0	Counter
uplink-total-pkts-pdn-rat-5g-nr	Int64	0	Counter
uplink-total-bytes-pdn-rat-5g-nr	Int64	0	Counter
uplink-total-pkts-dropped-pdn-rat-5g-nr	Int64	0	Counter
uplink-total-bytes-dropped-pdn-rat-5g-nr	Int64	0	Counter
downlink-total-pkts-pdn-rat-5g-nr	Int64	0	Counter
downlink-total-bytes-pdn-rat-5g-nr	Int64	0	Counter
downlink-total-pkts-dropped-pdn-rat-5g-nr	Int64	0	Counter
downlink-total-bytes-dropped-pdn-rat-5g-nr	Int64	0	Counter
uplink-total-pkts-pdn-rat-wlan	Int64	0	Counter
uplink-total-bytes-pdn-rat-wlan	Int64	0	Counter
uplink-total-pkts-dropped-pdn-rat-wlan	Int64	0	Counter
uplink-total-bytes-dropped-pdn-rat-wlan	Int64	0	Counter
downlink-total-pkts-pdn-rat-wlan	Int64	0	Counter
downlink-total-bytes-pdn-rat-wlan	Int64	0	Counter
downlink-total-pkts-dropped-pdn-rat-wlan	Int64	0	Counter
downlink-total-bytes-dropped-pdn-rat-wlan	Int64	0	Counter
uplink-total-pkts-pdn-rat-unknown	Int64	0	Counter
uplink-total-bytes-pdn-rat-unknown	Int64	0	Counter
uplink-total-pkts-dropped-pdn-rat-unknown	Int64	0	Counter
uplink-total-bytes-dropped-pdn-rat-unknown	Int64	0	Counter
downlink-total-pkts-pdn-rat-unknown	Int64	0	Counter
downlink-total-bytes-pdn-rat-unknown	Int64	0	Counter
downlink-total-pkts-dropped-pdn-rat-unknown	Int64	0	Counter
downlink-total-bytes-dropped-pdn-rat-unknown	Int64	0	Counter

Subscriber Level CLI

Use the following subscriber level CLIs as required:

show subscribers user-plane-only full all
show subscribers user-plane-only full msid msid_value
show subscribers user-plane-only { seid seid_value | callid callid_value } pdr all
show subscribers user-plane-only { seid seid_value | callid callid_value } far all
show subscribers user-plane-only { seid seid_value | callid callid_value } qer all
show subscribers user-plane-only { seid seid_value | callid callid_value } urr all
show subscribers user-plane-only { seid seid_value | callid callid_value } bar all
show subscribers user-plane-only { seid seid_value | callid callid_value } pdr full all
show subscribers user-plane-only { seid seid_value | callid callid_value } urr full all
show subscribers user-plane-only { seid seid_value | callid callid_value } far full all
show subscribers user-plane-only { seid seid_value | callid callid_value } qer full all
show subscribers user-plane-only { seid seid_value | callid callid_value } bar full all
show subscribers user-plane-only { seid seid_value | callid callid_value } pdr id id_value
show subscribers user-plane-only { seid seid_value | callid callid_value } flows full
show subscribers user-plane-only { seid seid_value | callid callid_value } bli full all

show subscribers user-plane-only full msid msid_value

The show subscribers user-plane-only full msid msid_value command displays the specific user information at instance-level and session-level for data collection during troubleshooting.

The following is a sample output of this command:

show subscribers user-plane-only full msid 404005123456789

  Local SEID      : [0x0004000000000000] 1125899906842624
  Remote SEID     : [0x0004000000000007] 1125899906842631
State           : Connected
  Connect Time    : Thu Sep 15 20:40:09 2022
  Idle time       : 00h11m25s
  Access Type: uplane-ipv4               Network Type: IP
  user-plane-service-name: UP21
  active-service-scheme-name: 
  Callid: 00004e21
  Rulebase: cisco
  Interface Type: Sxb                    
  eMPS Session: No
  eMPS Session Priority: 0
  Card/Cpu: 1/0                          Sessmgr Instance: 1
  IP address: 11.11.12.0
  Next Hop Ip Address: 
  Source context: EPC2-UP                Destination context: ISP1-UP
  PDN-Instance: starent.com         
  User-plane-Sx-addr: 20.20.20.106                 
  Control-plane-Sx-addr: 20.20.20.101                 
  Number of associated PDRs : 0
  Number of associated ADC PDRs : 0
  Number of associated FARs : 0
  Number of associated QERs : 1
  Number of associated BARs : 0
  Number of associated URRs : 0
  Uplink APN AMBR (bps) : 200000       Downlink APN AMBR (bps) : 200000
  active input acl: IPV4ACL            active output acl: IPV4ACL        
  active input ipv6 acl: n/a             active output ipv6 acl: n/a            
  Bandwidth Policy: bw_policy1
  FW-and-NAT Policy: n/a
  FW-and-NAT Policy ID: n/a
  Firewall Policy IPv4: n/a
  Firewall Policy IPv6: n/a
  NAT Policy NAT44: n/a
  NAT Policy NAT64: n/a
  Converged Session: No                 Converged Peer Callid:      n/a
  Visited Call: No                 
  Subscriber Parameters:
  IMSI: 404005123456789
  IMEI: 112233445566778
  MSISDN: 9890098900
  Charging Characteristics: 0x3412
  Rat Type: 6
  MCC MNC: 123765
  SGSN Address: 20.20.20.61
  ULI: TAI=12546312ab00
  Congestion Level: 0
  GGSN Address: 20.20.20.101
  User-Name: 9890098900@starent.com
  Session-ID: 1414146500000008
  input pkts: 0                                   output pkts: 0           
  input bytes: 0                                  output bytes: 0           
  input bytes dropped: 0                          output bytes dropped: 0         
  input pkts dropped: 0                           output pkts dropped: 0         
  CF Buffered Uplink Packets: 0                   CF Buffered Downlink Packets: 0           
  CF Buffered Uplink Bytes: 0                     CF Buffered Downlink Bytes: 0           
  Uplink Packets in Buffer: 0                     Downlink Packets in Buffer: 0           
  Buff Over-limit Uplink Pkts: 0                  Buff Over-limit Downlink Pkts: 0           
  DDN buffered pkts : 0                           DDN buffered bytes : 0           
  DDN buffer overflow drop pkts : 0               DDN buffer overflow drop bytes : 0           
  pk rate from user(bps): 0                       pk rate to user(bps): 0           
  ave rate from user(bps): 0                      ave rate to user(bps): 0           
  sust rate from user(bps): 0                     sust rate to user(bps): 0           
  pk rate from user(pps): 0                       pk rate to user(pps): 0           
  ave rate from user(pps): 0                      ave rate to user(pps): 0           
  sust rate from user(pps): 0                     sust rate to user(pps): 0           
  ipv4 bad hdr: 0                                 ipv4 ttl exceeded: 0         
ipv4 fragments sent: 0                          ipv4 could not fragment: 0         
  ipv4 bad length trim: 0         
  input pkts dropped (0 mbr): 0                   output pkts dropped (0 mbr): 0         
  ipv4 input acl drop: 0                          ipv4 output acl drop: 0         
  ipv6 input acl drop: 0                          ipv6 output acl drop: 0         
  ip source violations: 0                         ipv4 output no-flow drop: 0         
  ipv6 bad hdr: 0                                 ipv6 bad length trim: 0         
  ipv4 icmp packets dropped: 0         
  APN AMBR Uplink Pkts Drop: 0                     APN AMBR Downlink Pkts Drop: 0         
  APN AMBR Uplink Bytes Drop: 0                    APN AMBR Downlink Bytes Drop: 0         
  APN AMBR Uplink Pkts IP pref lowered: 0          APN AMBR Downlink Pkts IP pref lowered: 0         
  APN AMBR Uplink Bytes IP pref lowered: 0         APN AMBR Downlink Bytes IP pref lowered: 0         
  ITC Uplink Pkts Drop: 0                          ITC Downlink Pkts Drop: 0         
  ITC Uplink Bytes Drop: 0                         ITC Downlink Bytes Drop: 0         
  ITC Uplink Pkts IP pref lowered: 0               ITC Downlink Pkts IP pref lowered: 0         
  ITC Uplink Bytes IP pref lowered: 0              ITC Downlink Bytes IP pref lowered: 0         
  ITC Terminated Flows:: 0                         ITC Redirected Flows: 0         
  Flow Action Terminated Flows: 0                  Flow Action Redirected Flows: 0         
  ToS marked Uplink Pkts: 0                        ToS marked Downlink Pkts: 0         
  CC Dropped Uplink Pkts: 0                        CC Dropped Downlink Pkts: 0         
  CC Dropped Uplink bytes: 0                       CC Dropped Downlink Bytes: 0         
  Uplink Inflight Pkts: 0                          Downlink Inflight Pkts: 0            
  QFI Mismatch Uplink Pkts Drop: 0                 QFI Mismatch Downlink Pkts Drop: 0         
  Total QoS-Group(s) Active: 0            
  DNS-to-EDNS Uplink Pkts: 0                     DNS-to-EDNS Uplink Bytes: 0         
  EDNS Response Received: 0         

  Flow information:
    Current Active Flows:
      TCP: 0
      UDP: 0
    Total Flows:
      TCP: 0
      UDP: 0
      FP:  0

Static & Predef Rule Match stats:
Rule Name            Pkts-Down  Bytes-Down Pkts-Up    Bytes-Up   Hits       Match-Bypassed  FP-Down(Pkts/Bytes)  FP-Up(Pkts/Bytes)
-------------------- ---------- ---------- ---------- ---------- ---------- --------------  -------------------  -----------------

Dynamic Rule Match stats:
PDR Id   Rule Name            Pkts-Down  Bytes-Down Pkts-Up    Bytes-Up   Hits       Match-Bypassed  FP-Down(Pkts/Bytes)  FP-Up(Pkts/Bytes)
-------- -------------------- ---------- ---------- ---------- ---------- ---------- --------------  -------------------  -----------------

Post-Processing Rule Match stats:
Rule Name            Pkts-Down  Bytes-Down Pkts-Up    Bytes-Up   Hits       Match-Bypassed  FP-Down(Pkts/Bytes)  FP-Up(Pkts/Bytes)
-------------------- ---------- ---------- ---------- ---------- ---------- --------------  -------------------  -----------------

Firewall-NAT Access Rule Match stats:
Firewall-Ruledef Name Pkts-Down Bytes-Down    Pkts-Up   Bytes-Up Dr-Pkts-Dn Dr-Byts-Dn Dr-Pkts-Up Dr-Byts-Up       Hits
-------------------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------

QoS-Group Statistics:
QGR Name             Pkts-Down  Bytes-Down Pkts-Up    Bytes-Up   Hits       Match-Bypassed  FP-Down(Pkts/Bytes)  FP-Up(Pkts/Bytes)
-------------------- ---------- ---------- ---------- ---------- ---------- --------------  -------------------  -----------------

  CIoT APN Rate Control:
    Allowed UL Limit:   0                         Allowed DL limit:    0           
    Remaining UL Limit: 0                         Remaining DL limit:  0           
    Allowed Time unit: unrestricted               Status Validity Time: N/A              

  Local SEID      : [0x0004000000000002] 1125899906842626
  Remote SEID     : [0x0004000000000009] 1125899906842633
  State           : Connected
  Connect Time    : Thu Sep 15 20:51:03 2022
Idle time       : 00h00m31s
  Access Type: uplane-ipv4               Network Type: IP
  user-plane-service-name: UP21
  active-service-scheme-name: 
  Callid: 00004e23
  Rulebase: cisco
  Interface Type: Sxb                    
  eMPS Session: No
  eMPS Session Priority: 0
  Card/Cpu: 1/0                          Sessmgr Instance: 1
  IP address: 11.11.12.1
  Next Hop Ip Address: 
  Source context: EPC2-UP                Destination context: ISP1-UP
  PDN-Instance: starent.com         
  User-plane-Sx-addr: 20.20.20.106                 
  Control-plane-Sx-addr: 20.20.20.101                 
  Number of associated PDRs : 0
  Number of associated ADC PDRs : 0
  Number of associated FARs : 0
  Number of associated QERs : 1
  Number of associated BARs : 0
  Number of associated URRs : 0
  Uplink APN AMBR (bps) : 200000       Downlink APN AMBR (bps) : 200000
  active input acl: IPV4ACL            active output acl: IPV4ACL        
  active input ipv6 acl: n/a             active output ipv6 acl: n/a            
  Bandwidth Policy: bw_policy1
  FW-and-NAT Policy: n/a
  FW-and-NAT Policy ID: n/a
  Firewall Policy IPv4: n/a
  Firewall Policy IPv6: n/a
  NAT Policy NAT44: n/a
  NAT Policy NAT64: n/a
  Converged Session: No                 Converged Peer Callid:      n/a
  Visited Call: No                 
  Subscriber Parameters:
  IMSI: 404005123456789
  IMEI: 112233445566778
  MSISDN: 9890098900
  Charging Characteristics: 0x3412
  Rat Type: 6
  MCC MNC: 123765
  SGSN Address: 20.20.20.61
  ULI: TAI=12546312ab00
  Congestion Level: 0
  GGSN Address: 20.20.20.101
  User-Name: 9890098900@starent.com
  Session-ID: 141414650000000A
  input pkts: 0                                   output pkts: 0           
  input bytes: 0                                  output bytes: 0           
  input bytes dropped: 0                          output bytes dropped: 0         
  input pkts dropped: 0                           output pkts dropped: 0         
  CF Buffered Uplink Packets: 0                   CF Buffered Downlink Packets: 0           
  CF Buffered Uplink Bytes: 0                     CF Buffered Downlink Bytes: 0           
  Uplink Packets in Buffer: 0                     Downlink Packets in Buffer: 0           
  Buff Over-limit Uplink Pkts: 0                  Buff Over-limit Downlink Pkts: 0           
  DDN buffered pkts : 0                           DDN buffered bytes : 0           
  DDN buffer overflow drop pkts : 0               DDN buffer overflow drop bytes : 0           
  pk rate from user(bps): 0                       pk rate to user(bps): 0           
  ave rate from user(bps): 0                      ave rate to user(bps): 0           
  sust rate from user(bps): 0                     sust rate to user(bps): 0           
  pk rate from user(pps): 0                       pk rate to user(pps): 0           
  ave rate from user(pps): 0                      ave rate to user(pps): 0           
  sust rate from user(pps): 0                     sust rate to user(pps): 0           
  ipv4 bad hdr: 0                                 ipv4 ttl exceeded: 0         
  ipv4 fragments sent: 0                          ipv4 could not fragment: 0         
  ipv4 bad length trim: 0       
  input pkts dropped (0 mbr): 0                   output pkts dropped (0 mbr): 0         
  ipv4 input acl drop: 0                          ipv4 output acl drop: 0         
  ipv6 input acl drop: 0                          ipv6 output acl drop: 0         
  ip source violations: 0                         ipv4 output no-flow drop: 0         
  ipv6 bad hdr: 0                                 ipv6 bad length trim: 0         
  ipv4 icmp packets dropped: 0         
  APN AMBR Uplink Pkts Drop: 0                     APN AMBR Downlink Pkts Drop: 0         
  APN AMBR Uplink Bytes Drop: 0                    APN AMBR Downlink Bytes Drop: 0         
  APN AMBR Uplink Pkts IP pref lowered: 0          APN AMBR Downlink Pkts IP pref lowered: 0         
  APN AMBR Uplink Bytes IP pref lowered: 0         APN AMBR Downlink Bytes IP pref lowered: 0         
  ITC Uplink Pkts Drop: 0                          ITC Downlink Pkts Drop: 0         
  ITC Uplink Bytes Drop: 0                         ITC Downlink Bytes Drop: 0         
  ITC Uplink Pkts IP pref lowered: 0               ITC Downlink Pkts IP pref lowered: 0         
  ITC Uplink Bytes IP pref lowered: 0              ITC Downlink Bytes IP pref lowered: 0         
  ITC Terminated Flows:: 0                         ITC Redirected Flows: 0         
  Flow Action Terminated Flows: 0                  Flow Action Redirected Flows: 0         
  ToS marked Uplink Pkts: 0                        ToS marked Downlink Pkts: 0         
  CC Dropped Uplink Pkts: 0                        CC Dropped Downlink Pkts: 0         
  CC Dropped Uplink bytes: 0                       CC Dropped Downlink Bytes: 0         
  Uplink Inflight Pkts: 0                          Downlink Inflight Pkts: 0            
  QFI Mismatch Uplink Pkts Drop: 0                 QFI Mismatch Downlink Pkts Drop: 0         
  Total QoS-Group(s) Active: 0            
  DNS-to-EDNS Uplink Pkts: 0                     DNS-to-EDNS Uplink Bytes: 0         
  EDNS Response Received: 0         

  Flow information:
    Current Active Flows:
      TCP: 0
      UDP: 0
    Total Flows:
      TCP: 0
      UDP: 0
      FP:  0

Static & Predef Rule Match stats:
Rule Name            Pkts-Down  Bytes-Down Pkts-Up    Bytes-Up   Hits       Match-Bypassed  FP-Down(Pkts/Bytes)  FP-Up(Pkts/Bytes)
-------------------- ---------- ---------- ---------- ---------- ---------- --------------  -------------------  -----------------

Dynamic Rule Match stats:
PDR Id   Rule Name            Pkts-Down  Bytes-Down Pkts-Up    Bytes-Up   Hits       Match-Bypassed  FP-Down(Pkts/Bytes)  FP-Up(Pkts/Bytes)
-------- -------------------- ---------- ---------- ---------- ---------- ---------- --------------  -------------------  -----------------

Post-Processing Rule Match stats:
Rule Name            Pkts-Down  Bytes-Down Pkts-Up    Bytes-Up   Hits       Match-Bypassed  FP-Down(Pkts/Bytes)  FP-Up(Pkts/Bytes)
-------------------- ---------- ---------- ---------- ---------- ---------- --------------  -------------------  -----------------

Firewall-NAT Access Rule Match stats:
Firewall-Ruledef Name Pkts-Down Bytes-Down    Pkts-Up   Bytes-Up Dr-Pkts-Dn Dr-Byts-Dn Dr-Pkts-Up Dr-Byts-Up       Hits
-------------------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------

QoS-Group Statistics:
QGR Name             Pkts-Down  Bytes-Down Pkts-Up    Bytes-Up   Hits       Match-Bypassed  FP-Down(Pkts/Bytes)  FP-Up(Pkts/Bytes)
-------------------- ---------- ---------- ---------- ---------- ---------- --------------  -------------------  -----------------

  CIoT APN Rate Control:
    Allowed UL Limit:   0                         Allowed DL limit:    0           
    Remaining UL Limit: 0                         Remaining DL limit:  0           
    Allowed Time unit: unrestricted               Status Validity Time: N/A              

Total subscribers matching specified criteria: 1

VPP Statistics

To determine if the flows are offloaded to VPP, check for Fastpath statistics in the output of the following CLI commands:

show user-plane-service statistics all
show user-plane-service statistics analyzer name ip [ verbose ]
show user-plane-service statistics analyzer name ipv6 [ verbose ]
show user-plane-service statistics analyzer name tcp [ verbose ]
show user-plane-service statistics analyzer name udp [ verbose ]
show user-plane-service statistics analyzer name http [ verbose ]
show user-plane-service statistics analyzer name rtp [ verbose ]
show subscribers user-plane-only full callid call_id
show subscribers user-plane-only callid callid_value drop-statistics
show user-plane-service statistics drop-counter
show subscribers user-plane-only callid callid_value flows full
show subscribers user-plane-only callid callid_value flows flow-id flow_id

show subscribers user-plane-only callid callid_value drop-statistics

The show subscribers user-plane-only callid callid_value drop-statistics command displays the packet drop statistics with the respective cause (drop reason) at instance-level and session-level.

The following is a sample output of this command:

[local]UPF1# show subscribers user-plane-only callid 00004e21 drop-statistics
         
         Callid: 00004e21
         Interface Type: <interface-type>
         
         Packet Drop Data Statistics:
         -----------------------------------------------------
         NAT packets processing failure
            NAT on demand handling:                   0
            ICMP Packet translation:                  0
         FIREWALL packets processing failure
            Policy not found:                         0
         No Matching GX rule found:                   0
         Flow apply action
            Discard:                                  0
            Readdress Failure:                        0
        Packet exceeds the MTU size:                  0
        Failure in processing FAR Buffer packets:     0
        FAR Apply Action Drop:                        0
        Traffic Steering Failure:                     0
        QER Gate Status Closed:                       0
        Content-filtering Discard Action:             0
        IP Header Validation Failed:                  0
        ADF level failure
            DL TFT mismatch:                          0
        URL Blacklisting Discard Action:              0
        QGR Flow Action SGQ Discard:                  0
        QGR Policer Drops:                            0

Total subscribers matching specified criteria: 1
[local]UPF1#

show user-plane-service statistics drop-counter

The show user-plane-service statistics drop-counter command displays the packet drop statistics with the respective cause (drop reason) at instance-level and session-level.

The following is a sample output of this command:

[local]UPF1# show user-plane-service statistics drop-counter

         Packet Drop Data Statistics:
         -----------------------------------------------------
         NAT packets processing failure
            NAT on demand handling:                      0
            IP allocation is in progress:                0
            ICMP Packet translation:                     0
         FIREWALL packets processing failure
            Policy not found:                            0
         No Matching GX rule found:                      0
         Flow apply action
            Discard:                                     0
            Readdress Failure:                           0
         Packet exceeds the MTU size:                    0
         Failure in processing FAR Buffer packets:       0
         FAR Apply Action Drop:                          0
         Traffic Steering Failure:                       0
         QER Gate Status Closed:                         0
         Content-filtering Discard Action:               0
         IP Header Validation Failed:                    0
         ADF level failure
           UL TEID/QFI key mismatch:                     0
           DL TFT mismatch:                              0
           DL QFI mismatch:                              0
         URL Blacklisting Discard Action:                0
         DDN buffer overflow drop packets:               0
         APN AMBR Packets Drop:                          0
         ITC Packets Drop:                               0
         ACL Drop:                                       0
         CC Dropped Packets:                             0
         FastPath Misc Drops
             Overload Protection:                        0
             Invalid Client:                             0
             Stream ID 0:                                0
             Invalid Stream ID:                          0

NOTES:

For the show user-plane-service statistics drop-counter CLI command, the counters for the number of packets dropped due to various reasons will not be recovered post sessmgr recovery or ICSR (Inter-Chassis Session Recovery).
For the show subscribers user-plane-only callid callid_value drop-statistics CLI command, the counters for the number of packets dropped due to various reasons will be recovered post sessmgr recovery or ICSR.
The commands will not duplicate the counters that are captured at the session-level and instance-level.
The packets dropped due to "Quota Exhaust FAR Apply Action Drop" will be accumulated under "CC Dropped Packets" and not under the "FAR Apply Action Drop" counter.
The packets dropped under "Failure in processing FAR buffered packets" can be due to "no rule match", "teid not found", or other reasons. The respective counter will be incremented for such packets.

show subscribers user-plane-only callid callid_value flows full

The show subscribers user-plane-only callid callid_value flows full command displays the detailed information at the flow level which are required for debugging purposes.

The following is a sample output of this command:

[local]GVK-HUPF# show subscribers user-plane-only callid 017dc662 flows full
Tuesday November 22 18:35:35 UTC 2022

Callid: 017dc662
Interface Type: N4
IP address: n/a

Flow ID: 6:2
Uplink pkts: 0 Downlink pkts: 1
Uplink bytes: 0 Downlink bytes: 1040
Fast Path Info:
Uplink pkts: 0 Downlink pkts: 0
Uplink bytes: 0 Downlink bytes: 0
Total pkts: 0
Uplink Dropped pkts: 0 Downlink Dropped pkts: 4
Uplink Stream ID: 0x0 Downlink Stream ID: 0x1000007
Uplink Stream State: n/a Downlink Stream State: Passive
Client-ID : C6
UE IP address: 12.1.0.3 UE Port: 1001
Server IP address: 100.100.100.10 Server Port: 1001
Protocol: TCP
Service Chain Name: NA
Uplink Sfp Id: NA
Downlink Sfp Id: NA

Flow ID: 6:3
Uplink pkts: 5 Downlink pkts: 2
Uplink bytes: 200 Downlink bytes: 2080
Fast Path Info:
Uplink pkts: 0 Downlink pkts: 3
Uplink bytes: 0 Downlink bytes: 3120
Total pkts: 3
Uplink Dropped pkts: 0 Downlink Dropped pkts: 0
Uplink Stream ID: 0x0 Downlink Stream ID: 0x2000008
Uplink Stream State: n/a Downlink Stream State: Active
Client-ID : C6
UE IP address: 12.1.0.3 UE Port: 1000
Server IP address: 100.100.100.10 Server Port: 1000
Protocol: TCP
Service Chain Name: NA
Uplink Sfp Id: NA
Downlink Sfp Id: NA

Total Number of Active flows : 2

show subscribers user-plane-only callid callid_value flows flow-id flow_id

The show subscribers user-plane-only callid callid_value flows flow-id flow_id command displays the detailed information at the flow level which are required for debugging purposes.

The following is a sample output of this command:

# show sub user-plane-only callid 00004e22 flows flow-id 1:4

  Callid: 00004e22
  Interface Type: Sxb
  IP address: n/a

  Flow ID: 1:4
  Uplink pkts: 17                                 Downlink pkts: 0
  Uplink bytes: 1026                              Downlink bytes: 0
  Fast Path Info:
   Uplink pkts: 0                                 Downlink pkts: 0
   Uplink bytes: 0                                Downlink bytes: 0
   Total pkts: 0
   Uplink Dropped pkts: 0                         Downlink Dropped pkts: 0
   Uplink Stream ID: 0x0000008                    Downlink Stream ID: 0x0
   Uplink Stream State: Passive                   Downlink Stream State: n/a
   Client-ID : C2
  UE IP address: 12.2.0.13                        UE Port: 1007
  Server IP address: 100.100.100.10               Server Port: 1005
  Protocol: TCP
  Service Chain Name: NA
  Uplink Sfp Id: NA
  Downlink Sfp Id: NA

Total Number of Active flows : 1

SNMP Support

The system uses the Simple Network Management Protocol (SNMP) to send traps or events to the EMS server or an alarm server on the network. You must configure SNMP settings to communicate with those devices.

The SNMP MIB Reference describes the MIBs and SNMP traps that are supported by UPF and StarOS.

The following SNMP traps are available in support of their respective feature or functionality:

N4 Session/Node Level Reporting Procedure

The following traps are available to track status and conditions GTP-U path failure:

EGTPUPathFailure: This trap is generated when no response is received for GTP-U ECHO requests and data path failure is detected toward peer EPC Node.
EGTPUPathFailureClear: This trap is generated when the data path toward the peer node is available.

UP Session Recovery

The following traps are available after session recovery in the User Plane node:

ManagerFailure: This trap is generated when there is failure in the Software manager.
TaskFailed: This trap is generated when a noncritical task has failed and the appropriate recovery steps begin.
TaskRestart: This trap is generated when a noncritical task has restarted after an earlier failure.
SessMgrRecoveryComplete: This trap is generated when Session Manager recovery completes. This is typically caused by the failure of Session Manager task and successful completion of recovery.
ManagerRestart: This trap is generated when the identified manager task has been restarted.

Sx Association

The following traps are available to track the status of an Sx Association:

SxPeerAssociated: This trap is triggered when an Sx association is detected.
SxPeerAssociationRelease: This trap is triggered when an Sx association release is detected.

URL Blockedlisting

The following SNMP trap are available in support of URL Blockedlisting feature:

BLDBError: Specifies the blockedlisting OPTBLDB file error that is displayed with an error code.
BLDBErrorClear: Specifies the blockedlisting OPTBLDB file error removed.
BLDBUpgradeError: Specifies the blockedlisting OPTBLDB file error displayed with an error code.
BLDBUpgradeErrorClear: Specifies the blockedlisting OPTBLDB file error removed.

Enabling SNMP Traps

Use the following configuration to enable an SNMP trap.

configure 
   snmp trap enable trap_name 
   end

For supplemental information about SNMP Support, see Management Settings chapter in the ASR 5500 System Administration Guide.

Troubleshooting UPF Features

N4 or Datapath

The following CLI commands are available for troubleshooting N4 or datapath related issues:

show gtpu statistics
show user-plane-service { all | bandwidth-policy | charging-action | edr-format | group-of-ruledefs | gtpp-group | name | pdn-instance | rulebase | ruledef | statistics | xheader-format }

NOTES:
- all : Displays all User Plane services.
- bandwidth-policy : Displays information for bandwidth-policy in User Plane service.
- charging-action : Displays information for Charging actions in User Plane service.
- edr-format : Displays information for EDR format in user Plane service.
- group-of-ruledefs : Displays information on Group of Ruledefs configured in User Plane service.
- gtpp-group : Displays information for bandwidth policy in User Plane service.
- name : Displays information for specific User Plane service name.
- pdn-instance : Displays information for PDN instance.
- rulebase : Displays information for rulebase in User Plane service.
- ruledef : Displays information for ruledef in User Plane service.
- statistics : Displays node-level statistics for User Plane.
  
  Additionally, you can also use: show user-plane-service statistics { all | analyzer | charging-action | fapi | rulebase | tethering-detection }
- xheader-format : Displays information for X-Header format in User Plane service.
show user-plane-service content-filtering category policy-id ( all | id id_value }
- content-filtering : Displays content filtering information.
- category : Displays content filtering category information.
- policy-id : Displays content filtering category Policy-ID and its definition.
- all : Displays definitions of all content filtering category policies.
- id id_value : Displays content filtering category definition of a particular Policy-ID. id_value is an integer ranging from 1 through 4,294,967,295.
show sx-service { all | name | statistics }

NOTES:
- all : Displays all Sx Services.
- name : Displays information for specific Sx Service name.
- statistics : Displays the total of collected information for specific protocol since last restart or clear command.

Content Filtering

Use the following CLI command for troubleshooting CF related issues:

In releases prior to 2022.01.0:

show user-plane-service inline-services { content-filtering | info | url-blacklisting }

From 2022.01.0 and later releases:

show user-plane-service inline-services { content-filtering | info | url-blockedlisting }

NOTES:

content-filtering : Displays content filtering information.
info : Displays information of inline services.
url-blockedlisting : Displays URL Blockedlisting parameters in User Plane service.

URL Blacklisting

Use the following CLI command for troubleshooting URL Blacklisting related issues: show user-plane-service url-blacklisting database { all | debug-only | facility | url }

NOTES:

all : Displays all URL Blacklisting database configurations.
debug-only : Displays the URL Blacklisting static database debug information.
facility : Displays URL Blacklisting database configuration per facility.
url : Displays particular database information for URL Blacklisting.

UCC 5G UPF Configuration and Administration Guide, Release 2023.04

Bias-Free Language

Book Title

UCC 5G UPF Configuration and Administration Guide, Release 2023.04

Chapter Title