Troubleshoot of User Browsing Issues in LTE

Introduction

This document describes the user data browsing issues on the 4G network.

Prerequisites

Cisco recommends that you have knowledge of these nodes' functionalities

1. Serving Packet Data Gateway (SPGW)

2. Control and User Plane Separation (CUPS)

Symptoms

Before you start with testing and log collection, you need to check these mentioned details:

1. Check issue is for which Packet Data Network (PDN) data type: IPv4/IPv6/IPv4v6

2. Check issue is with any particular Access Point Name (APN) or all the APNs because the issue can be related to specific APNs as well.

3. Check if the URL is an enterprise URL/customer app URL or some regular service URL and accordingly if the problem is with a specific VPN.

4. Check if the issue occurs when accessing the URL directly from the browser or while accessing the web app itself.

5. Is the issue intermittent in nature like post restart of handset/refresh web URLs start working or the issue is consistent and does not work even after handset restart?

6. Check the Rejection cause observed and for which rating group.

Logs Collection/Testing

Note: For these kinds of issues, you need to do real-time online troubleshooting with problematic user IMSI on which you need to collect logs/traces accordingly.

Before proceeding with the testing and log collection.

Flush the subscriber from the node and also clear browsing history/database from testing user handset so that it can freshly attach

clear subscriber imsi <IMSI number>  ------------------ to be executed in the node to clear the subscriber

1. Start with the testing for the subscriber with any PDN type.

2. Log the putty session and start monitor subscriber with verbosity 5 and enable this option.

SPGW:
Press + for times then it collects the logs verbosity 5 logs then select next options
+++++
S,X,A,Y,56,26,33,34,19,37,35,88,89
Once option 75 is pressed then select 3,4,8 then press esc

CUPS::
on CP:
monitor subscriber imsi <IMSI> +++++        S, X,A,Y,56,26,33,34,19,37,35,88,89
on UP:
monitor subscriber imsi <IMSI> +++++        S,X,A,Y,56,26,33,34,19,37,35,88,89

3. Please enable these debug logs and log the putty session and make sure the session must not be terminated (press tab/enter every few min so that session does not terminate).

On SPGW:
logging filter active facility sessmgr level debug
logging filter active facility acsmgr level debug
logging filter active facility npumgr-acl level debug
logging filter active facility firewall level debug
logging filter active facility vpn level debug
logging filter active facility vpnmgr level debug
logging active   ---------------- to enable the logging
no logging active ---------------- to disable the logging

On CP:
logging filter active facility sessmgr level debug
logging filter active facility sxdemux level debug
logging filter active facility firewall level debug
logging filter active facility vpn level debug
logging filter active facility vpnmgr level debug
logging active ---------------- to enable the logging
no logging active ---------------- to disable the logging

On UP:
logging filter active facility sessmgr level debug
logging filter active facility sxdemux level debug
logging filter active facility npumgr-acl level debug
logging filter active facility firewall level debug
logging filter active facility vpn level debug
logging filter active facility vpnmgr level debug
logging active ---------------- to enable the logging
no logging active ---------------- to disable the logging

Note :: These logging has to be enabled for short time depending on the CPU utilization because it 
increase the utilization so while enabling logging need to keep a watch on CPU

4. Configure mode, please enable logging monitor for the subscriber

config
logging monitor msid <imsi>
end

5. Attach the Subscriber and browse the URL continuously for 3 to 5 mins and while browsing execute this command multiple times and log the putty session for the same.

ON SPGW/SAEGW:
show subscriber full imsi <>
show active-charging session full imsi <>
show subscriber pgw-only full imsi <>
show subscriber sgw-only full imsi <>
show subscribers data-rate summary imsi <>
show ims-authorization sessions full imsi <>
show subscribers debug-info msid <>

On CP node:

 Show subscriber full imsi <imsi>
Show active-charging session full imsi <imsi>
show subscribers pgw-only full imsi <>
show subscribers sgw-only full imsi <>
show session subsystem facility sessmgr instance <> verbose
show logs

On UP node:
show sub user-plane-only full callid  <>
show sub user-plane-only callid <> urr full all
show sub user-plane-only callid <> far full all
show sub user-plane-only callid <> pdr full all
show subscribers user-plane-only callid <> far all
show subscribers user-plane-only callid <> far
show subs data-rate call <callid>
show subscribers user-plane-only flows
show user-plane-service statistics all
show user-plane-service statistic rulebase name <rulebase_name>

6. After 5 mins of browsing, execute  no logging active  in the terminal which is opened at Step 4

7. Disable the logging monitor for the subscriber.

Config
no logging monitor msid <imsi>

8. Execute this command to get the call id of the subscriber and log the putty session for this as well.

Show subscriber full imsi <imsi>.  --> to get the call id
show logs callid <call_id>
show logs

9. If the call id is present then it is clear that subscriber session logs were collected if not, needs to run again.

Analysis

1. Check if the DNS resolution is successful or not. If it is successful then there is no issue with DNS.

DNS Resolution Traces

2. Check the subscriber level statistics to review the packet drops.

SPGW/CP:
Show subscriber full imsi <imsi number>

CUPS UP:
show user-plane-only full imsi <>

input pkts: 455 output pkts: 474 
input bytes: 75227 output bytes: 103267 
input bytes dropped: 0 output bytes dropped: 0 
input pkts dropped: 0 output pkts dropped: 0 
input pkts dropped due to lorc : 0 output pkts dropped due to lorc : 0 
input bytes dropped due to lorc : 0 
in packet dropped suspended state: 0 out packet dropped suspended state: 0 
in bytes dropped suspended state: 0 out bytes dropped suspended state: 0 
in packet dropped sgw restoration state: 0 out packet dropped sgw restoration state: 0 
in bytes dropped sgw restoration state: 0 out bytes dropped sgw restoration state: 0 
pk rate from user(bps): 18547 pk rate to user(bps): 25330 
ave rate from user(bps): 6182 ave rate to user(bps): 8443 
sust rate from user(bps): 5687 sust rate to user(bps): 7768 
pk rate from user(pps): 13 pk rate to user(pps): 14 
ave rate from user(pps): 4 ave rate to user(pps): 4 
sust rate from user(pps): 4 sust rate to user(pps): 4 
link online/active percent: 92 
ipv4 bad hdr: 0 ipv4 ttl exceeded: 0 
ipv4 fragments sent: 0 ipv4 could not fragment: 0 
ipv4 input acl drop: 0 ipv4 output acl drop: 0 
ipv4 bad length trim: 0 
ipv6 input acl drop: 0 ipv6 output acl drop: 0 
ipv4 input css down drop: 0 ipv4 output css down drop: 0 
ipv4 input css down drop: 0 ipv4 output css down drop: 0 
ipv4 output xoff pkts drop: 0 ipv4 output xoff bytes drop: 0 
ipv6 output xoff pkts drop: 0 ipv6 output xoff bytes drop: 0 
ipv6 input ehrpd-access drop: 0 ipv6 output ehrpd-access drop: 0 
input pkts dropped (0 mbr): 0 output pkts dropped (0 mbr): 0 
ip source violations: 0 ipv4 output no-flow drop: 0 
ipv6 egress filtered: 0 
ipv4 proxy-dns redirect: 0 ipv4 proxy-dns pass-thru: 0 
ipv4 proxy-dns drop: 0 
ipv4 proxy-dns redirect tcp connection: 0 
ipv6 bad hdr: 0 ipv6 bad length trim: 0 
ip source violations no acct: 0 
ip source violations ignored: 0 
dormancy total: 0 handoff total: 0 
ipv4 icmp packets dropped: 0 
APN AMBR Input Pkts Drop: 0 APN AMBR Output Pkts Drop: 0 
APN AMBR Input Bytes Drop: 0 APN AMBR Output Bytes Drop: 0 
APN AMBR UE Overload Input Pkts Drop: 0 APN AMBR UE Overload Output Pkts Drop: 0 
APN AMBR UE Overload Input Bytes Drop: 0 APN AMBR UE Overload Output Bytes Drop: 0 
Access-flows:0
Num Auxiliary A10s:0

3. Check the show active charging command output for ECS/ACS level packet drop and check if there are any packet drops then check in the configuration what is the action configured.

Show active-charging session full imsi <imsi num> or show sub user-plane-only full callid  <>
 Ruledef Name Pkts-Down Bytes-Down Pkts-Up Bytes-Up Hits Match-Bypassed
-------------------- ---------- ---------- ---------- ---------- ---------- --------------
dns_free_covid 4 428 4 340 8 0
icmpv6 0 0 5 1423 5 0
ip-pkts 479 103670 432 74488 764 429    

4. Check the TCP connection is established successfully between UE and the server.

5. If there are no drops observed in any of these steps then there is no issue in the node.

Packet Drops

1. Check the subscriber release statistics to determine whether you are experiencing packet drops similar to those shown here.

 Total Dropped Packets : 132329995
Total Dropped Packet Bytes: 14250717212

Total PP Dropped Packets : 0
Total PP Dropped Packet Bytes: 0

R7Gx Rule-Matching Failure Stats:
Total Dropped Packets : 871921
Total Dropped Packet Bytes : 86859232

P2P random drop stats:
Total Dropped Packets : 0
Total Dropped Packet Bytes : 0

2. Check the percentage of failures observed in the show subscriber output. If the packet drops are less than 1%, it is most likely a fluke and does not have any effect.

input pkts: 455 output pkts: 474 
input bytes: 75227 output bytes: 103267 
input bytes dropped: 0 output bytes dropped: 0 
input pkts dropped: 0 output pkts dropped: 0 

3. If you notice packet drops in the RX rating group and ITC packet drops, this is most likely due to a bandwidth issue and the subscriber package expired.

 ITC Packets Drop:                           47235019

4. At the ECS level, it is important to verify the DPI configuration, including the rule definition, charging action, and rulebase, to determine if any blocking factors exist. There are various types of drops at the ECS level, and the next course of action depends on the specific type of drop encountered.

5. MTU size for the packet size which is passing and not processed.

6. Intermediate path issues where the packet is getting dropped can be identified from TCP dump/user-level traces.

The recovery action plan is not the same for this type of issue as it varies as per the pattern of the issue.