Troubleshoot Diameter Error Code: "IP-CAN_SESSION_NOT_AVAILABLE (5065)" in CPS

Introduction

This document describes various investigation methods to handle the Diameter Error Code: "IP-CAN_SESSION_NOT_AVAILABLE (5065)" in Cisco Policy Suite (CPS).

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Linux
• CPS
• Policy Control Charge Rule Function (PCRF)
• Diameter

Note: Cisco recommends that you must have privilege root access to CPS CLI.

Components Used

The information in this document is based on CPS 21.1.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

Diameter Error Code: "IP-CAN_SESSION_NOT_AVAILABLE (5065)" is a commonly seen Diameter Experimental-Result-Code in PCRF which handles Voice over Long-Term Evolution (VoLTE) services.

This error is not generated for a healthy VoLTE system. There are a few scenarios that are discussed in this document where this error is reported and its various approaches to troubleshoot them.

In general, If the PCRF fails to do a session bind between Rx and Gx sessions, then it responds to the Application Function (AF) with an AA-Answer, that includes an Experimental-Result-Code Attribute Value Pair (AVP) set to the value " IP-CAN_SESSION_NOT_AVAILABLE (5065) ".

Problem

Scenario 1

Difference in Framed-IP-Prefix AVP Value in GX_ (CCR-I) and Rx_AAR

The basic rule for Rx to Gx Session bind is based on the same Framed-IPv6-Prefix received from Gx_Credit Control Request Initial (CCR-I) 543 while the Gx session creation and the one received from Rx_Authorization Authentication Request (AAR) for Rx Session creation.

In this scenario, it is observed that all the error samples of IP-CAN_SESSION_NOT_AVAILABLE (5065) are reported from a specific list of Framed-IPv6-Prefix. These Framed-IPv6-Prefixes are not reported in Gx_CCR-I .

This CLI provides the unique list and total count of Framed-IPv6-Prefixfor which the IP-CAN_SESSION_NOT_AVAILABLE (5065) is reverted by CPS to AF.

[root@LAB-PCRF02CO-pcrfclient01 broadhop]# zgrep -B80 "IP-CAN_SESSION_NOT_AVAILABLE" consolidated-engine.*.gz | grep "Framed-IPv6-Prefix:" | cut -f3,4,5 -d: | sort| uniq -c
      6  2402:XXXX:YYYY
      3  2402:PPPP:QQQQ
      1  2402:JJJJ:KKKK

This CLI provides the unique list and count of Framed-IPv6-Prefix reported in Gx_CCR-I from Packet Data Network Gateway (PGW) .

[root@LAB-PCRF02CO-pcrfclient01 broadhop]# zgrep -A 50 CCR-I consolidated-engine.*.gz | grep Framed-IPv6-Prefix: | cut -f 3,4,5 -d: | sort | uniq -c
    115  2402:AAAA:BBBB
    116  2402:NNNN:MMMM
    246  2402:SSSS:RRRR

Note: For the CLI execution, you must log in to pcrfclient01 of CPS and navigate to /var/log/broadhop/ directory.

The expectation for a healthy system is, the Framed-IPv6-Prefixreceived from Gx_CCR-I and Rx_AAR be from the same IP prefix pool allocated for Gm IP (PGW to AF).

In this scenario, it is clear that, for all the failure samples collected, the Framed-IP-Prefix reported in Rx_AAR is completely different from the Framed-IP-Prefix received in Gx_CCR-I.

Scenario 2

Contradiction in Gx Stale Session Timer and Session Expiration Timer Set in CPS

1. What is the Gx Stale session timer?

This timer is armed every time a message is received or sent for any given Gx session. When the timer expires, a Gx_Re-Auth-Request (RAR)  is sent with Re-Auth-Request-Type AVP set to AUTHORIZE_ONLY (0) for that Gx session towards PGW. If a Gx_Re-Auth-Answer (RAA) is received with Result-Code AVP value set to DIAMETER_UNKNOWN_SESSION_ID (5002) or DIAMETER_UNABLE_TO_COMPLY (5012) the Gx session is deemed stale and removed from the PCRF internal database. On any activity over the Gx interface (Gx_RAR/Gx_CCR) , the timer is reset.

2. What is a Session Expiration timer?

If no messages are received within the timer set, the session is removed from PCRF.

In the problematic scenario, the timers are set like this. 

Gx Stale session timer: 180 mins

Session expiration time: 240 min

As per the timer configuration, after 180 mins a Keep alive RAR is generated and sent to PGW, and if the session is inactive for 240 mins, the session gets removed from PCRF DB.

If the Keep alive RAR is missed, due to multiple reasons [Gx interface flap, PGW out of service, or any corner case], the session reaches its expiration time, and at the same time if there is any Rx_AAR received from AF, you can see this error, and a Gx_RAR is sent to PGW with  Session-Release-Cause: UNSPECIFIED_REASON (0).

INFO   : (core) Deletion started at Wed Oct 19 07:07:03 IST 2022
WARN   : (core) Resending original message since the existing session is in deletion and this session triggered a start

Message: com.broadhop.diameter2.messages.DiameterRequestMessage
                Timestamp: 2022-10-19 01:37:03,676
                Application Id: Gx (16777238)
                Command Code: Gx_RAR (258)
                Dest host: 0006-diamproxy.lab.cisco.com
                Dest realm: vpcrf.cisco.com
                Device protocol: GX_TGPP
                End to end id: 0 (0x00000000)
                Hop by hop id: 0 (0x00000000)
                Origin state: 0
                Stack name: null
                Origin realm: x.x.x.x
                Session-Id: 0006-diamproxy.lab.cisco.com;1200190897;28719897;634ebb76-11902
                Auth-Application-Id: 16777238
                Re-Auth-Request-Type: AUTHORIZE_ONLY (0)
                Session-Release-Cause: UNSPECIFIED_REASON (0)

Also, the Gx session is marked for soft delete at PCRF. Since PCRF failed to bind the Gx session, a Rx_AAA is responded back with Experimental-Result-Code AVP set to the value  IP-CAN_SESSION_NOT_AVAILABLE (5065).

This is the complete call flow that explains the scenario:

=========================================================================
LAB-PCRF02-qns05  [2022-10-19 07:07:03,676] ===============================================
POLICY RESULT SUCCESS:
        session action = Delete
        domainId = IMS
        locationId = ims
        SERVICES: IMS_DEFAULT
        TRIGGER: Message: com.broadhop.diameter2.messages.DiameterRequestMessage
                Timestamp: 2022-10-19 01:37:03,673
                Application Id: Rx (16777236)
                Command Code: Rx_AAR (265)
                Dest realm: x.x.x.x
                Device protocol: RX_TGPP
                End to end id: 1274920388 (0x4bfdbdc4)
                Hop by hop id: 3440610237 (0xcd138bbd)
                Origin state: 0
                Stack name: null
                Origin host: x.x.x.x
                Origin realm: x.x.x.x
                Session-Id: x.x.x.x;168440;1;5.3576663599.2;pcsf.ims322-002
                Auth-Application-Id: 16777236
                Media-Component-Description:
                                Media-Component-Number: 1
                                Media-Sub-Component:
                                                Flow-Number: 1
                                                Flow-Description: permit in 17 from xxxx:xxxx:xxxx:xxxx:xxxx:xxxx  to xxxx:xxxx:xxxx:xxxx:xxxx:xxxx
                                                Flow-Description: permit out 17 from xxxx:xxxx:xxxx:xxxx:xxxx:xxxx to xxxx:xxxx:xxxx:xxxx:xxxx:xxxx
                                Media-Sub-Component:
                                                Flow-Number: 2
                                                Flow-Description: permit in 17 from xxxx:xxxx:xxxx:xxxx:xxxx:xxxx  to xxxx:xxxx:xxxx:xxxx:xxxx:xxxx
                                                Flow-Description: permit out 17 from xxxx:xxxx:xxxx:xxxx:xxxx:xxxx to xxxx:xxxx:xxxx:xxxx:xxxx:xxxx
                                                Flow-Usage: RTCP (1)
                                Codec-Data: downlink\r\noffer\r\nm=audio 23096 RTP/AVP 108 102 8 18 100 101 116\r\na=rtpmap:108 AMR/8000\r\na=fmtp:108 mode-set=0,2,4,7; mode-change-period=2;                                        Codec-Data: uplink\r\nanswer\r\nm=audio 41660 RTP/AVP 108 116\r\na=rtpmap:108 AMR/8000/1\r\na=ptime:20\r\na=maxptime:240\r\na=rtpmap:116 telephone-event/
                                Media-Type: AUDIO (0)
                                Max-Requested-Bandwidth-UL: 37000
                                Max-Requested-Bandwidth-DL: 37000
                                Flow-Status: ENABLED (2)
                                RS-Bandwidth: 462
                                RR-Bandwidth: 1387
                Subscription-Id:
                                Subscription-Id-Type: END_USER_SIP_URI (2)
                                Subscription-Id-Data: sip:+91XXXXXXXXXX
                AF-Charging-Identifier: ABCA4C23EA4C0320221019772
                Specific-Action: INDICATION_OF_LOSS_OF_BEARER (2)
                Specific-Action: INDICATION_OF_RECOVERY_OF_BEARER (3)
                Specific-Action: INDICATION_OF_RELEASE_OF_BEARER (4)
                Specific-Action: IP_CAN_CHANGE (6)
                Framed-IPv6-Prefix: XXXX:XXXX:XXXX:XXXX (0xxxxxxxxxxxxxxxxxxxxx)
                Supported-Features:
                                Vendor-Id: 10415
                                Feature-List-ID: 1
                                Feature-List: 65587
                Service-Info-Status: FINAL SERVICE INFORMATION (0)
                Req-Received-Time: 1666143423673
                QPS-Internal-Route-Record-Host: x.x.x.x
                QPS-Internal-Route-Record-Realm: x.x.x.x
        DEBUG MSGS:
                INFO   : (core) Tagging message with ID: RX_TGPP
                INFO   : (core) Successful load by secondary key: FramedIpv6PrefixKey:framedIpv6Prefix:XXXX%XXXX%XXXX%XXXX
                INFO   : (core) Start session triggered
                INFO   : (core) Deletion started at Wed Oct 19 07:07:03 IST 2022
                WARN   : (core) Resending original message since the existing session is in deletion and this session triggered a start
                INFO   : (core) domain: IMS
                INFO   : (gx) Sending RAR Message on session 0006-diamproxy.lab.cisco.com;1200190897;28719897;634ebb76-11902
                INFO   : (core) Adding network-id keys for missing credentials
                INFO   : (custrefdata) QueryTable: TAI_BASED_POLICY Field: {ECGI=XXXXX, packname=-1, AN GW IPV4=*, RAT_TYPE=1004, IMSI_Prefix=XXXXX, TAI=4042052329, APN=XXX}
                INFO   : (custrefdata) QueryTable: LAC_BASED_POLICY Field: {MCC_MNC=XXXX, packname=-1, rat=1004, CGI=*, SGSN_IP=*, CI=*, IMSI_Prefix=XXXXXX, LAC=XXXX, APN=XX}
                INFO   : (core) domain: IMS
                INFO   : (core) Session removed
        SERVICE CALC MSGS:
        USE CASES:
                INFO   : (use-cases) Use case ‘Lab IMS Default', status: true, Condition: No Condition Set
        SENT MESSAGES (asynchronous):
        Message: com.broadhop.diameter2.messages.DiameterRequestMessage
                Timestamp: 2022-10-19 01:37:03,676
                Application Id: Gx (16777238)
                Command Code: Gx_RAR (258)
                Dest host: 0006-diamproxy.lab-spgw01.cisco.com
                Dest realm: vpcrf.cisco.com
                Device protocol: GX_TGPP
                End to end id: 0 (0x00000000)
                Hop by hop id: 0 (0x00000000)
                Origin state: 0
                Stack name: null
                Origin realm: x.x.x.x
                Session-Id: 0006-diamproxy.lab-spgw01.cisco.com;1200190897;28719897;634ebb76-11902
                Auth-Application-Id: 16777238
                Re-Auth-Request-Type: AUTHORIZE_ONLY (0)
                Session-Release-Cause: UNSPECIFIED_REASON (0)
=========================================================================

At the same time, for the same packet, another session action: create  is processed by CPS but couldn’t bind the Gx session, and hence a 5065 Error is reverted to AF.

LAB-VPCRF02-qns05  [2022-10-19 07:07:03,696] ===============================================
POLICY RESULT SUCCESS:
        session action = Create
        TRIGGER: Message: com.broadhop.diameter2.messages.DiameterRequestMessage
                Timestamp: 2022-10-19 01:37:03,673
                Application Id: Rx (16777236)
                Command Code: Rx_AAR (265)
                Dest realm: x.x.x.x
                Device protocol: RX_TGPP
                End to end id: 1274920388 (0x4bfdbdc4)
                Hop by hop id: 3440610237 (0xcd138bbd)
                Origin state: 0
                Stack name: null
                Origin host: x.x.x.x
                Origin realm: x.x.x.x
                Session-Id: x.x.x.x;168440;1;5.3576663599.2;pcsf.ims322-002
                Auth-Application-Id: 16777236
                Media-Component-Description:
                                Media-Component-Number: 1
                                Media-Sub-Component:
                                                Flow-Number: 1
                                                Flow-Description: permit in 17 from xxxxxx 8975:b101 41660 toxxxxxx::2 23096
                                                Flow-Description: permit out 17 fromxxxxxx::2 23096 toxxxxxx:b101 41660
                                Media-Sub-Component:
                                                Flow-Number: 2
                                                Flow-Description: permit in 17 from xxxxxx 8975:b101 41660 toxxxxxx::2 23096
                                                Flow-Description: permit out 17 fromxxxxxx::2 23096 toxxxxxx:b101 41660
                                                Flow-Usage: RTCP (1)
                                Codec-Data: downlink\r\noffer\r\nm=audio 23096 RTP/AVP 108 102 8 18 100 101 116\r\na=rtpmap:108 AMR/8000\r\na=fmtp:108 mode-set=0,2,4,7; mode-change-period=2;                                        Codec-Data: uplink\r\nanswer\r\nm=audio 41660 RTP/AVP 108 116\r\na=rtpmap:108 AMR/8000/1\r\na=ptime:20\r\na=maxptime:240\r\na=rtpmap:116 telephone-event/                                                 Media-Type: AUDIO (0)
                                Max-Requested-Bandwidth-UL: 37000
                                Max-Requested-Bandwidth-DL: 37000
                                Flow-Status: ENABLED (2)
                                RS-Bandwidth: 462
                                RR-Bandwidth: 1387
                Subscription-Id:
                                Subscription-Id-Type: END_USER_SIP_URI (2)
                                Subscription-Id-Data: sip:+xxxxxxxx
                AF-Charging-Identifier: ABCA4C23EA4C0320221019772
                Specific-Action: INDICATION_OF_LOSS_OF_BEARER (2)
                Specific-Action: INDICATION_OF_RECOVERY_OF_BEARER (3)
                Specific-Action: INDICATION_OF_RELEASE_OF_BEARER (4)
                Specific-Action: IP_CAN_CHANGE (6)
                Framed-IPv6-Prefix: xxxx:xxxx:xxxx:xxxx
                Supported-Features:
                                Vendor-Id: 10415
                                Feature-List-ID: 1
                                Feature-List: 65587
                Service-Info-Status: FINAL SERVICE INFORMATION (0)
                Req-Received-Time: 1666143423673
                QPS-Internal-Route-Record-Host: x.x.x.x
                QPS-Internal-Route-Record-Realm: x.x.x.x
        DEBUG MSGS:
                INFO   : (core) Tagging message with ID: RX_TGPP
                INFO   : (rx) Could not perform Gx session binding for Framed-IP-Address null, Framed-IPv6-Prefix xxxx:xxxx:xxxx:xxxx
                INFO   : (core) Start session triggered
                INFO   : (rx) Could not perform Gx session binding for Framed-IP-Address null, Framed-IPv6-Prefix xxxx:xxxx:xxxx:xxxx
                INFO   : (rx) Sending 265 Message on session x.x.x.x;168440;1;5.3576663599.2;pcsf.ims322-002
                ERROR  : (rules) Could not perform Gx session binding: Could not perform Gx session binding 5065
        SERVICE CALC MSGS:
        USE CASES:
        SENT MESSAGES (asynchronous):
        Message: com.broadhop.diameter2.messages.DiameterResponseMessage
                Timestamp: 2022-10-19 01:37:03,696
                Application Id: Rx (16777236)
                Command Code: Rx_AAA (265)
                Dest host: x.x.x.x
                Dest realm: x.x.x.x
                Device protocol: RX_TGPP
                End to end id: 1274920388 (0x4bfdbdc4)
                Hop by hop id: 3440610237 (0xcd138bbd)
                Origin state: 0
                Stack name: null
                Origin realm: x.x.x.x
                Session-Id: x.x.x.x;168440;1;5.3576663599.2;pcsf.ims322-002
                Auth-Application-Id: 16777236
                QPS-Internal-Route-Record-Host: x.x.x.x
                QPS-Internal-Route-Record-Realm: x.x.x.x
                Experimental-Result:
                        Vendor-Id: 10415
                        Experimental-Result-Code: IP-CAN_SESSION_NOT_AVAILABLE (5065)

=========================================================================

Scenario 3

Subscriber Session Available at PCRF, but Not in PGW Caused Due to Gx Interface Flap

There is a corner case scenario, where the subscriber sessions are removed in PGW, but it is present in PCRF. This happens, when there is a Gx link flap caused by a transport layer break.

Failure call flow diagram:

Failure Scenario: IP_CAN_SESSION_NOT_AVAILABLE(5065)

Sample call flow for this error from the Engine log:

lab-qns07  [2022-08-24 00:42:48,643] ===============================================
POLICY RESULT SUCCESS:
session action = None
domainId = IMS
locationId = ims
SERVICES: IMS_DEFAULT
TRIGGER: Message: com.broadhop.diameter2.messages.DiameterRequestMessage
Timestamp: 2022-08-23 19:12:48,638
Application Id: Rx (16777236)
Command Code: Rx_AAR (265)
Dest realm: x.x.x.x
Device protocol: RX_TGPP
End to end id: 1797266185 (0x6b201b09)
Hop by hop id: 1690686712 (0x64c5d4f8)
Origin state: 0
Stack name: null
Origin host: x.x.x.x
Origin realm: x.x.x.x
Session-Id: x.x.x.x;367169;1;5.1236049266.2;pcsf-stdn.ims332-012
Auth-Application-Id: 16777236
Media-Component-Description:
Media-Component-Number: 1
Media-Sub-Component:
Flow-Number: 1
Flow-Description: permit in 17 from xxxxxx 50060 to xxxxxx::4 28532
Flow-Description: permit out 17 from xxxxxx::4 28532 to xxxxxxx 50060
Media-Sub-Component:
Flow-Number: 2
Flow-Description: permit in 17 from xxxxxx 50060 to xxxxxx::4 28532
Flow-Description: permit out 17 from xxxxxx::4 28532 to xxxxxxx 50060
Flow-Usage: RTCP (1)
Codec-Data: uplink\r\noffer\r\nm=audio 50060 RTP/AVP 104 102 105 96\r\na=rtpmap:104 AMR-WB/16000/1\r\na=fmtp:104 mode-set=0,3;mode-change-capability=2;max-
Codec-Data: downlink\r\nanswer\r\nm=audio 28532 RTP/AVP 102 96\r\na=rtpmap:102 AMR/8000/1\r\na=fmtp:102 mode-set=0,2,4,7; mode-change-period=2; mode-change-capability=2;
Media-Type: AUDIO (0)
Max-Requested-Bandwidth-UL: 72000
Max-Requested-Bandwidth-DL: 72000
Flow-Status: ENABLED (2)
RS-Bandwidth: 362
RR-Bandwidth: 1087
Subscription-Id:
Subscription-Id-Type: END_USER_SIP_URI (2)
Subscription-Id-Data: sip:+919XXXXXX
AF-Charging-Identifier: "PCSF:1-MU1-cfed-12-00000000630526b2-0000000017442828"
Specific-Action: INDICATION_OF_LOSS_OF_BEARER (2)
Specific-Action: INDICATION_OF_RECOVERY_OF_BEARER (3)
Specific-Action: INDICATION_OF_RELEASE_OF_BEARER (4)
Specific-Action: IP_CAN_CHANGE (6)
Framed-IPv6-Prefix: xxxx:xxxx:xxxx:xxxx
Supported-Features:
Vendor-Id: 10415
Feature-List-ID: 1
Feature-List: 65587
Service-Info-Status: FINAL SERVICE INFORMATION (0)
Req-Received-Time: 1661281963763
QPS-Internal-Route-Record-Host: x.x.x.x
QPS-Internal-Route-Record-Realm: x.x.x.x
DEBUG MSGS:
INFO   : (core) Tagging message with ID: RX_TGPP
INFO   : (core) Successful load by secondary key: FramedIpv6PrefixKey:framedIpv6Prefix:xxxx%3Axxxx%3Axxxx%3Axxxx
INFO   : (core) Start session triggered
INFO   : (rx) Success binding to Gx session 0001-diamproxy.ndc3-ims-lab-sbc1;1106437295;560189406;63051856-de02 by IP Address
INFO   : (TGPP) [Rel8, Rel9, Rel10, NetLoc, NetLocUntrustedWLAN] features supported for Vendor-Id 10415, Feature-List-ID 1
INFO   : (rx) Creating new diameter session x.x.x.x;367169;1;5.1236049266.2;pcsf-stdn.ims332-012
INFO   : (policy) Load CRD for Unknown Subscriber
INFO   : (location) Location found for avp matching: logical_apn\ims
WARN   : (core) Session has switched from known to unknown as subscriber could not be found
INFO   : (core) Adding unknown service code: IMS_DEFAULT for session
INFO   : (core) domain: IMS
INFO   : (core) Adding network-id keys for missing credentials
INFO   : (core) domain: IMS
INFO   : (rx) Sending AAA Message on session x.x.x.x;367169;1;5.1236049266.2;pcsf-stdn.ims332-012
INFO   : (gx) Installing preconfigured rule: _6_1_2_AF_AUDIO
INFO   : (gx) Installing preconfigured rule: _6_1_1_AF_AUDIO
INFO   : (et) Event trigger 2 added
INFO   : (et) Event trigger 5 added
INFO   : (et) Event trigger 6 added
INFO   : (et) Event trigger 7 added
INFO   : (gx) Sending RAR Message on session 0001-diamproxy.ndc3-ims-lab-sbc1;1106437295;560189406;63051856-de02
SERVICE CALC MSGS:
USE CASES
INFO   : (use-cases) Use case 'WLAN', status: false, Condition: ("Rat-Type WLAN"=false)
INFO   : (use-cases) Use case 'IMS Default', status: true, Condition: No Condition Set
SENT MESSAGES (asynchronous):
Message: com.broadhop.diameter2.messages.DiameterResponseMessage
Timestamp: 2022-08-23 19:12:48,643
Application Id: Rx (16777236)
Command Code: Rx_AAA (265)
Dest host: x.x.x.x
Dest realm: x.x.x.x
Device protocol: RX_TGPP
End to end id: 1797266185 (0x6b201b09)
Hop by hop id: 1690686712 (0x64c5d4f8)
Origin state: 0
Stack name: null
Origin realm: x.x.x.x
Session-Id: x.x.x.x;367169;1;5.1236049266.2;pcsf-stdn.ims332-012
Auth-Application-Id: 16777236
IP-CAN-Type: 3GPP_EPS (5)
RAT-Type: EUTRAN (1004)
Supported-Features:
Vendor-Id: 10415
Feature-List-ID: 1
Feature-List: 65587QPS-Internal-Route-Record-Host: x.x.x.x
QPS-Internal-Route-Record-Realm: x.x.x.x
Result-Code: DIAMETER_SUCCESS (2001)
Message: com.broadhop.diameter2.messages.DiameterRequestMessage
Timestamp: 2022-08-23 19:12:48,643
Application Id: Gx (16777238)
Command Code: Gx_RAR (258)
Dest host: 0001-diamproxy.ndc3-ims-lab-sbc1
Dest realm: vpcrf.cisco.com
Device protocol: GX_TGPP
End to end id: 0 (0x00000000)
Hop by hop id: 0 (0x00000000)
Origin state: 0
Stack name: null
Origin realm: x.x.x.x
Session-Id: 0001-diamproxy.ndc3-ims-airpcrf02-sbc1;1106437295;560189406;63051856-de02
Auth-Application-Id: 16777238
Re-Auth-Request-Type: AUTHORIZE_ONLY (0)
Charging-Rule-Install:
Charging-Rule-Definition:
Charging-Rule-Name: _6_1_2_AF_AUDIO
Rating-Group: 2
Flow-Information:
Flow-Description: permit in 17 from xxxxxx 50061 to xxxxxxx28533
Flow-Direction: UPLINK (2)
Flow-Information:
Flow-Description: permit out 17 from xxxxx 28533 to xxxxxx 50061
Flow-Direction: DOWNLINK (1)
Flow-Status: ENABLED (2)
QoS-Information:
QoS-Class-Identifier: Conversational (1)
Max-Requested-Bandwidth-UL: 1449
Max-Requested-Bandwidth-DL: 1449
Guaranteed-Bitrate-UL: 1449
Guaranteed-Bitrate-DL: 1449
Reporting-Level: RATING_GROUP_LEVEL (1)
Precedence: 0
AF-Charging-Identifier: "PCSF:1-MU1-cfed-12-00000000630526b2-0000000017442828"
Flows:
Media-Component-Number: 1
Flow-Number: 2
Charging-Rule-Install:
Charging-Rule-Definition:
Charging-Rule-Name: _6_1_1_AF_AUDIO
Rating-Group: 2
Flow-Information:
Flow-Description: permit in 17 from xxxxxx 50060 to xxxxxx 28532
Flow-Direction: UPLINK (2)
Flow-Information:
Flow-Description: permit out 17 from xxxxxxx 28532 to 2xxxxx 50060
Flow-Direction: DOWNLINK (1)
Flow-Status: ENABLED (2)
QoS-Information:
QoS-Class-Identifier: Conversational (1)
Max-Requested-Bandwidth-UL: 72000
Max-Requested-Bandwidth-DL: 72000
Guaranteed-Bitrate-UL: 72000
Guaranteed-Bitrate-DL: 72000
Reporting-Level: RATING_GROUP_LEVEL (1)
Precedence: 0
AF-Charging-Identifier: "PCSF:1-MU1-cfed-12-00000000630526b2-0000000017442828"
Flows:
Media-Component-Number: 1
Flow-Number: 1
Event-Trigger: RAT_CHANGE (2)
Event-Trigger: LOSS_OF_BEARER (5)
Event-Trigger: RECOVERY_OF_BEARER (6)
Event-Trigger: IP-CAN_CHANGE (7)
=======================================================================


lab-qns04  [2022-08-24 00:42:49,849] ===============================================
POLICY RESULT SUCCESS:
session action = None
domainId = IMS
locationId = ims
SERVICES: IMS_DEFAULT
TRIGGER: Message: com.broadhop.diameter2.messages.DiameterResponseMessage
Timestamp: 2022-08-23 19:12:49,844
Application Id: Gx (16777238)
Command Code: Gx_RAA (258)
Device protocol: GX_TGPP
End to end id: 2244560894 (0x85c947fe)
Hop by hop id: 2040440255 (0x799ea5bf)
Origin state: 1657916833
Stack name: null
Origin host: 001-diamproxy.ndc3-ims-lab-sbc1
Origin realm: vpcrf.cisco.com
Session-Id: 0001-diamproxy.ndc3-ims-lab-sbc1;1106437295;560189406;63051856-de02
Origin-State-Id: 1657916833
Error-Message: Unknown Session ID
Req-Received-Time: 0
Result-Code: DIAMETER_UNKNOWN_SESSION_ID (5002)
DEBUG MSGS:
INFO   : (core) Tagging message with ID: GX_TGPP
INFO   : (core) Successful load by primary key: diameterSessionKey:0001-diamproxy.ndc3-ims-airpcrf02-sbc1%3B1106437295%3B560189406%3B63051856-de02
INFO   : (core) Stop session triggered
INFO   : (core) Deletion started at Wed Aug 24 00:42:49 IST 2022
INFO   : (core) Soft deletion triggered for session
INFO   : (policy) Load CRD for Unknown Subscriber
INFO   : (location) Location found for avp matching: logical_apn\ims
WARN   : (core) Session has switched from known to unknown as subscriber could not be found
INFO   : (core) Adding unknown service code: IMS_DEFAULT for session
INFO   : (core) domain: IMS
INFO   : (core) Adding network-id keys for missing credentials
INFO   : (core) domain: IMS
SERVICE CALC MSGS:
USE CASES:
INFO   : (use-cases) Use case 'WLAN', status: false, Condition: ("Rat-Type WLAN"=false)
INFO   : (use-cases) Use case 'Vodafone IMS Default', status: true, Condition: No Condition Set

=========================================================================

lab-qns04  [2022-08-24 00:43:02,349] ===============================================
POLICY RESULT SUCCESS:
session action = Create
TRIGGER: Message: com.broadhop.diameter2.messages.DiameterRequestMessage
Timestamp: 2022-08-23 19:13:02,330
Application Id: Rx (16777236)
Command Code: Rx_AAR (265)
Dest realm: x.x.x.x
Device protocol: RX_TGPP
End to end id: 1811425453 (0x6bf828ad)
Hop by hop id: 1690687469 (0x64c5d7ed)
Origin state: 0
Stack name: null
Origin host: x.x.x.x
Origin realm: x.x.x.x
Session-Id: x.x.x.x;367195;1;5.1236262316.2;pcsf-stdn.ims332-012
Auth-Application-Id: 16777236
Media-Component-Description:
Media-Component-Number: 1
Media-Sub-Component:
Flow-Number: 1
Flow-Description: permit in 17 from xxxx:xxxx:xxxx:xxxx: 50032 to xxxx:xxxx:xxxx:xxxx::6 5144
Flow-Description: permit out 17 from xxxx:xxxx:xxxx:xxxx::6 xxxx to xxxx:xxxx:xxxx:xxxx: 50032
Media-Sub-Component:
Flow-Number: 2
Flow-Description: permit in 17 from 2402:3a80:2027:2db8:ac0e:ff4:dc53:af02 50033 to 2402:3a80:2000:606::6 5145
Flow-Description: permit out 17 from 2402:3a80:2000:606::6 5145 to 2402:3a80:2027:2db8:ac0e:ff4:dc53:af02 50033
Flow-Usage: RTCP (1)
Codec-Data: downlink\r\noffer\r\nm=audio 5144 RTP/AVP 96 8 18 100 101 97\r\na=rtpmap:96 AMR/8000\r\na=fmtp:96 mode-set=0,2,4,7; mode-change-period=2; mode-change-Codec-Data: uplink\r\nanswer\r\nm=audio 50032 RTP/AVP 96 97\r\na=rtpmap:96 AMR/8000/1\r\na=fmtp:96 mode-set=0,2,4,7;mode-change-capability=2;max-red=0\r\na=rtpmap:97 Media-Type: AUDIO (0)
Max-Requested-Bandwidth-UL: 37000
Max-Requested-Bandwidth-DL: 37000
Flow-Status: ENABLED (2)
RS-Bandwidth: 0
RR-Bandwidth: 0
Subscription-Id:
Subscription-Id-Type: END_USER_SIP_URI (2)
Subscription-Id-Data: sip:+xxxxxxx
AF-Charging-Identifier: GYlkAABklmEFGKS7UELMBecR
Specific-Action: INDICATION_OF_LOSS_OF_BEARER (2)
Specific-Action: INDICATION_OF_RECOVERY_OF_BEARER (3)
Specific-Action: INDICATION_OF_RELEASE_OF_BEARER (4)
Specific-Action: IP_CAN_CHANGE (6)
Framed-IPv6-Prefix: xxxxxxxxx
Supported-Features:
Vendor-Id: 10415
Feature-List-ID: 1
Feature-List: 65587
Service-Info-Status: FINAL SERVICE INFORMATION (0)
Req-Received-Time: 1661281976257
QPS-Internal-Route-Record-Host: x.x.x.x
QPS-Internal-Route-Record-Realm: x.x.x.x
DEBUG MSGS:
INFO   : (core) Tagging message with ID: RX_TGPP
INFO   : (rx) Could not perform Gx session binding for Framed-IP-Address null, Framed-IPv6-Prefix xxxxxx
INFO   : (core) Start session triggered
INFO   : (rx) Could not perform Gx session binding for Framed-IP-Address null, Framed-IPv6-Prefix xxxxxx
INFO   : (rx) Sending 265 Message on session x.x.x.x;367195;1;5.1236262316.2;pcsf-stdn.ims332-012
ERROR  : (rules) Could not perform Gx session binding: Could not perform Gx session binding 5065
SERVICE CALC MSGS:
USE CASES:
SENT MESSAGES (asynchronous):
Message: com.broadhop.diameter2.messages.DiameterResponseMessage
Timestamp: 2022-08-23 19:13:02,349
Application Id: Rx (16777236)
Command Code: Rx_AAA (265)
Dest host: x.x.x.x
Dest realm: x.x.x.x
Device protocol: RX_TGPP
End to end id: 1811425453 (0x6bf828ad)
Hop by hop id: 1690687469 (0x64c5d7ed)
Origin state: 0
Stack name: null
Origin realm: x.x.x.x
Session-Id: x.x.x.x;367195;1;5.1236262316.2;pcsf-stdn.ims332-012
Auth-Application-Id: 16777236
QPS-Internal-Route-Record-Host: x.x.x.x
QPS-Internal-Route-Record-Realm: x.x.x.x
Experimental-Result:
Vendor-Id: 10415
Experimental-Result-Code: IP-CAN_SESSION_NOT_AVAILABLE (5065)


===============================================

When you closely check the Error KPI trend for RX_AAR [IP-CAN_SESSION_NOT_AVAILABLE (5065)]  and Gx_RAR [DIAMETER_UNKNOWN_SESSION_ID (5002)], you can see the linear increment of these errors after a Gx interface flap.

Gx_RAR & Rx_AAR Failure Trend

Solution

Approach for Scenario 1

The solution to this issue must be executed outside of CPS. You must perform a reconciliation of the IP pool used in the Gm interface (the interface between SPGW to AF), and ensure the same Framed-IP-Prefixes  are used in both PGW and AF for the User Equipment (UE).

Approach for Scenario 2

Set the session expiration timer to be > (3 * Gx stale-session timer).

Procedure to Set the Timer

Step 1. Login to Policy Builder with Read/Write user.

Step 2. Set the Default Gx Stale Session Timer Minutes  as the recommended value.

Gx Stale Session Timer Configuration

Step 3. Set the Session Expiration Hours and Session Expiration Minutes  as recommended 3 * of Gx Stale session timer.

Session Expiration Timer Configuration

Step 4. Save the Repository and publish to the latest version.

Approach for Scenario 3

Introduce a Rx_ASR towards AF to clear the already created Rx Session if CPS receives Diameter Error Code=5002 in the response of Gx_RAR from PGW. In this way, if the session is not present in PGW, and CPS receives Diameter Error 5002 , along with soft deletion of the Gx session, CPS sends a Rx_ASR message towards the AF node with an Abort-Cause: 0.

Procedure to Set the Rx_ASR Parameter

Step 1. Log in to the Cluster Manager as the root user.

Step 2. Backupqns.conf file.

cp /etc/broadhop/qns.conf /etc/broadhop/qns.conf.$(date +%m%d%Y).bkp

Step 3. Edit the file /etc/broadhop/qns.conf  with vi editors and insert the line “-Dsend.Rx.ASR.For.AdditionalErrorCodes=5002”

Step 4. Build puppet script.

/var/qps/install/current/puppet/modules/qps/templates/var/broadhop/build_etc.sh

Step 5. Populate the changes in all VMs.

/var/qps/install/current/scripts/upgrade/reinit.sh

Step 6. Restart all to get the changes effective.

restartall.sh

This is the expected call flow after the required parameter change.

Success Scenario After the Change

This is the expected call flow from CPS engine log after the required parameter change.

=========================================================================
[2022-10-10 09:46:28,899] ===============================================
POLICY RESULT SUCCESS:
session action = None
domainId = F1660RxPriorityLevel
locationId = F1660IMS
SERVICES: RxPriorityLevel
TRIGGER: Message: com.broadhop.diameter2.messages.DiameterRequestMessage
Timestamp: 2022-10-10 09:46:28,332
Application Id: Rx (16777236)
Command Code: Rx_AAR (265)
Dest host: qns-site-server-1
Dest realm: cisco.com
Device protocol: RX_TGPP
End to end id: 535285358 (0x1fe7ce6e)
Hop by hop id: 655749306 (0x2715f0ba)
Origin state: 0
Stack name: null
Origin host: site-host-rx
Origin realm: site-rx-client
Session-Id: site-host-rx;132419;581;311482316600001IMSSession
Auth-Application-Id: 16777236
Media-Component-Description:
Media-Component-Number: 1
Media-Sub-Component:
Flow-Number: 1
Flow-Description: permit in ip from xxxxxxx 10000 to xxxxxxx:1 20000
Flow-Description: permit out ip from xxxxxxxxx1 20000 to xxxxxxx 10000
Flow-Usage: AF_SIGNALLING (2)
AF-Application-Identifier: lab.test.f1660.afapid01
Media-Type: VIDEO (1)
Max-Requested-Bandwidth-UL: 50000
Max-Requested-Bandwidth-DL: 50000
RS-Bandwidth: 800
RR-Bandwidth: 800
Specific-Action: INDICATION_OF_FAILED_RESOURCES_ALLOCATION (9)
Specific-Action: CHARGING_CORRELATION_EXCHANGE (1)
Specific-Action: INDICATION_OF_LOSS_OF_BEARER (2)
Specific-Action: INDICATION_OF_RECOVERY_OF_BEARER (3)
Specific-Action: INDICATION_OF_RELEASE_OF_BEARER (4)
Specific-Action: IP_CAN_CHANGE (6)
Specific-Action: ACCESS_NETWORK_INFO_REPORT (12)
Subscription-Id:
Subscription-Id-Type: END_USER_SIP_URI (2)
Subscription-Id-Data: sip:xxxxxxx
Framed-IP-Address: x.x.x.x (0x01020304)
Supported-Features:
Vendor-Id: 10415
Feature-List-ID: 1
Feature-List: 32
MIP6-Agent-Info:
MIP-Home-Agent-Host:
Destination-Realm: pcef
Destination-Host: vzr
Req-Received-Time: 1665395188271
QPS-Internal-Route-Record-Host: site-host-rx
QPS-Internal-Route-Record-Realm: site-rx-client
DEBUG MSGS:
INFO   : (core) Tagging message with ID: RX_TGPP
INFO   : (diameter)  Adding extracted AVP to policy: DiameterMessageAvp [code=Media-Type, value=1]
INFO   : (core) Successful load by secondary key: FramedIpKey:framedIp:x.x.x.x
INFO   : (core) Start session triggered
INFO   : (rx) Success binding to Gx session site-host-gx;132419;575;9001660001 by IP Address
INFO   : (TGPP) [NetLoc] features supported for Vendor-Id 10415, Feature-List-ID 1
INFO   : (rx) Creating new diameter session site-host-rx;132419;581;311482316600001IMSSession
INFO   : (gx) Installing preconfigured rule: _1_1_1_AF_VIDEO
INFO   : (et) Event trigger 2 added
INFO   : (et) Event trigger 5 added
INFO   : (et) Event trigger 6 added
INFO   : (et) Event trigger 7 added
INFO   : (et) Event trigger 28 added
INFO   : (gx) Sending RAR Message on session site-host-gx;132419;575;9001660001
SERVICE CALC MSGS:
USE CASES:
INFO   : (use-cases) Use case 'F1660_Rx_priorityLevel', status: true, Condition: No Condition Set
SENT MESSAGES (asynchronous):
com.broadhop.diameter2.actions.ISendDiameterResponse
DestinationName:
ResponseMessage: Message: com.broadhop.diameter2.messages.DiameterResponseMessage
Timestamp: 2022-10-10 09:46:28,893
Application Id: Rx (16777236)
Command Code: Rx_AAA (265)
Dest host: site-host-rx
Dest realm: site-rx-client
Device protocol: RX_TGPP
End to end id: 535285358 (0x1fe7ce6e)
Hop by hop id: 655749306 (0x2715f0ba)
Origin state: 0
Stack name: null
Origin realm: cisco
Session-Id: site-host-rx;132419;581;311482316600001IMSSession
Auth-Application-Id: 16777236
IP-CAN-Type: 3GPP_EPS (5)
RAT-Type: EUTRAN (1004)
Supported-Features:
Vendor-Id: 10415
Feature-List-ID: 1
Feature-List: 32
QPS-Internal-Route-Record-Host: site-host-rx
QPS-Internal-Route-Record-Realm: site-rx-client
Result-Code: DIAMETER_SUCCESS (2001)


com.broadhop.diameter2.actions.ISendDiameterRequest
RequestMessage: Message: com.broadhop.diameter2.messages.DiameterRequestMessage
Timestamp: 2022-10-10 09:46:28,899
Application Id: Gx (16777238)
Command Code: Gx_RAR (258)
Dest host: site-host-gx
Dest realm: site-gx-client
Device protocol: GX_TGPP
End to end id: 0 (0x00000000)
Hop by hop id: 0 (0x00000000)
Origin state: 0
Stack name: null
Session-Id: site-host-gx;132419;575;9001660001
Auth-Application-Id: 16777238
Re-Auth-Request-Type: 0
Charging-Rule-Install:
Charging-Correlation-Indicator: 0
Charging-Rule-Definition:
Charging-Rule-Name: _1_1_1_AF_VIDEO
Flow-Information:
Flow-Description: permit in ip from xxxx:xxxx:xxxx:xxxx: 10000 toxxxx:xxxx:xxxx:xxxx 20000
Flow-Information:
Flow-Description:permit in ip from xxxx:xxxx:xxxx:xxxx: 10000 toxxxx:xxxx:xxxx:xxxx 20000
QoS-Information:
QoS-Class-Identifier: 3
Max-Requested-Bandwidth-UL: 204800
Max-Requested-Bandwidth-DL: 204800
Allocation-Retention-Priority:
Priority-Level: 1
Pre-emption-Capability: 0
Pre-emption-Vulnerability: 1
Event-Trigger: 2
Event-Trigger: 5
Event-Trigger: 6
Event-Trigger: 7
Event-Trigger: 28
DestinationName: lab-1:diameter-lb
=========================================================================

[2022-10-10 09:46:29,323] ===============================================
POLICY RESULT SUCCESS:
session action = None
domainId = F1660RxPriorityLevel
locationId = F1660IMS
SERVICES: RxPriorityLevel
TRIGGER: Message: com.broadhop.diameter2.messages.DiameterResponseMessage
Timestamp: 2022-10-10 09:46:29,148
Application Id: Gx (16777238)
Command Code: Gx_RAA (258)
Device protocol: GX_TGPP
End to end id: 2577399809 (0x99a00001)
Hop by hop id: 1095304429 (0x414904ed)
Origin state: 0
Stack name: null
Origin host: site-host-gx
Origin realm: site-gx-client
Session-Id: site-host-gx;132419;575;9001660001
Auth-Application-Id: 16777238
Req-Received-Time: 0
 Result-Code: DIAMETER_UNKNOWN_SESSION_ID (5002)
DEBUG MSGS:
INFO   : (core) Tagging message with ID: GX_TGPP
INFO   : (core) Lock obtained on key: diameterSessionKey:site-host-gx%3B132419%3B575%3B9001660001
INFO   : (core) Successful load by primary key: diameterSessionKey:site-host-gx%3B132419%3B575%3B9001660001
INFO   : (core) Stop session triggered
INFO   : (core) Deletion started at Mon Oct 10 09:46:29 UTC 2022
INFO   : (core) Soft deletion triggered for session
INFO   : (rx) Trigger Rx-ASR for error code 5002
INFO   : (rx) Sending ASR Message on session site-host-rx;132419;581;311482316600001IMSSession
SERVICE CALC MSGS:
USE CASES:
INFO   : (use-cases) Use case 'F1660_Rx_priorityLevel', status: true, Condition: No Condition Set
SENT MESSAGES (asynchronous):
com.broadhop.diameter2.actions.ISendDiameterRequest
RequestMessage: Message: com.broadhop.diameter2.messages.DiameterRequestMessage
Timestamp: 2022-10-10 09:46:29,323
Application Id: Rx (16777236)
Command Code: Rx_ASR (274)
Dest host: site-host-rx
Dest realm: site-rx-client
Device protocol: RX_TGPP
End to end id: 0 (0x00000000)
Hop by hop id: 0 (0x00000000)
Origin state: 0
Stack name: null
Session-Id: site-host-rx;132419;581;311482316600001IMSSession
Auth-Application-Id: 16777236
Abort-Cause: 0
DestinationName: lab-1:diameter-lb

=========================================================================