Determine the Impact of the Issue

• Is the issue affecting subscriber experience?

• Is the issue affecting billing?

• Is the issue affecting all subscribers?

• Is the issue affecting all subscribers on a specific service?

• Is there anything else common to the issue?

• Have there been any changes performed on the CPS system or any other systems?

• Has there been an increase in subscribers?

• Initially, categorize the issue to determine the level of support needed.

Important

Use the dump_utility.py script to collect useful troubleshooting information for Cisco technical support. Information is printed on the terminal and stored in a log file: /var/tmp/dumputility-<date_time_when_executed>.log. For more information about this utility, refer to the list of CPS Commands in the CPS Operations Guide.

This section covers the following two commands:

• trace.sh

• trace_id.sh

For more information on trace commands, refer to Policy Tracing and Execution Analyzer section in CPS Operations Guide.

• Run the following command on pcrfclient01 and verify that all the processes are reported as Running.

  For CPS 7.0.0 and higher releases:

  /var/qps/bin/control/statusall.sh

  Program 'cpu_load_trap' 
    status                            Waiting 
    monitoring status                 Waiting 
  Process 'collectd' 
    status                            Running 
    monitoring status                 Monitored 
    uptime                            42d 17h 23m 
  Process 'auditrpms.sh' 
    status                            Running 
    monitoring status                 Monitored 
    uptime                            28d 20h 26m 
  System 'qns01' 
    status                            Running 
    monitoring status                 Monitored 
  The Monit daemon 5.5 uptime: 21d 10h 26m 
  Process 'snmpd' 
    status                            Running 
    monitoring status                 Monitored 
    uptime                            21d 10h 26m 
  Process 'qns-1' 
    status                            Running 
    monitoring status                 Monitored 
    uptime                            6d 17h 9m 

• Run /var/qps/bin/diag/diagnostics.sh command on pcrfclient01 and verify that no errors/failures are reported in output.

  /var/qps/bin/diag/diagnostics.sh 
  CPS Diagnostics HA Multi-Node Environment 
  --------------------------- 
  Ping check for all VMs... 
  Hosts that are not 'pingable' are added to the IGNORED_HOSTS variable...[PASS                                                         ] 
  Checking basic ports for all VMs...[PASS] 
  Checking qns passwordless logins for all VMs...[PASS] 
  Checking disk space for all VMs...[PASS] 
  Checking swap space for all VMs...[PASS] 
  Checking for clock skew for all VMs...[PASS] 
  Checking CPS diagnostics... 
    Retrieving diagnostics from qns01:9045...[PASS] 
    Retrieving diagnostics from qns02:9045...[PASS] 
    Retrieving diagnostics from qns03:9045...[PASS] 
    Retrieving diagnostics from qns04:9045...[PASS] 
    Retrieving diagnostics from pcrfclient01:9045...[PASS] 
    Retrieving diagnostics from pcrfclient02:9045...[PASS] 
  Checking svn sync status between pcrfclient01 & 02... 
  svn is not sync between pcrfclient01 & pcrfclient02...[FAIL] 
  Corrective Action(s): Run ssh pcrfclient01 /var/qps/bin/support/recover_svn_sync.sh 
  Checking HAProxy statistics and ports... 

  For more information on diagnostics.sh, refer to diagnostics.sh section in CPS Operations Guide.

• Perform the following actions to verify VMs status is reported as UP and healthy and no alarms are generated for any VMs.

  • Login to the VMware console

  • Verify the VM statistics, graphs and alarms through the console.

• Verify if any trap is generated by CPS.

  Note	/var/log/snmp/trap file is updated on active load balancer (LB) only whenever the trap is generated.

  cd /var/log/snmp

  tailf trap

• Verify if any error is reported in CPS logs.

  cd /var/log/broadhop

  grep -i error consolidated-qns.log

  grep -i error consolidated-engine.log

• Monitor the following KPIs on Grafana for any abnormal behavior:

  • CPU usage of all instances on all the VMs

  • Memory usage of all instances on all VMs

  • Free disk space on all instances on all VMs

  • Diameter messages load: CCR-I, CCR-U, CCR-T, AAR, RAR, STR, ASR, SDR

  • Diameter messages response time: CCR-I, CCR-U, CCR-T, AAR, RAR, STR, ASR, SDA

• Errors for diameter messages.

  Run the following command on pcrfclient01:

  tailcons | grep diameter | grep -i error

• Response time for sessionmgr insert/update/delete/query.

  • Average read, write, and total time per sec:

    mongotop --host sessionmgr* --port port_number

  • For requests taking more than 100ms:

    SSH to sessionmgr VMs:

    tailf /var/log/mongodb-<portnumber>.log

    Note	Above commands will by default display requests taking more than 100 ms, until and unless the following parameter has been configured on mongod process --slows XYZms. XYZ represents the value in milliseconds desired by user.

• Garbage collection.

  Check the service-qns-*.log from all policy server (QNS), load balancer (lb) and PCRF VMs. In the logs look for “GC” or “FULL GC”.

• Session count.

  Run the following command on pcrfclient01:

  session_cache_ops.sh --count

• Run the following command on pcrfclient01 and verify that the response time is under expected value and there are no errors reported.

  /opt/broadhop/qns-1/control/top_qps.sh

• Use the following command to check mongoDB statistics on queries/inserts/updates/deletes for all CPS databases (and on all primary and secondary databases) and verify if there are any abnormalities (for example, high number of insert/update/delete considering TPS, large number of queries going to other site).

  mongostat --host <sessionmgr VM name> --port <dBportnumber>

  For example,

  mongostat --host sessionmgr01 --port 27717

• Use the following command for all CPS databases and verify if there is any high usage reported in output. Here considering session database as an example:

  mongotop --host <sessionmgr VM name> --port <dBportnumber>

  For example,

  mongotop --host sessionmgr01 --port 27717

• Verify EDRs are getting generated by checking count of entries in CDR database.

• Verify EDRs are getting replicated by checking count of entries in the databases.

• Determine most recently inserted CDR record in MySQL database and compare the insert time with the time the CDR was generated. Time difference should be within 2 min or otherwise signifies lag in replication.

• Count of CCR-I/CCR-U/CCR-T/RAR messages from/to GW.

• Count of failed CCR-I/CCR-U/CCR-T/RAR messages from/to GW. If GW has capability, capture details at error code level.

  Run the following command on pcrfclient01:

  cd /var/broadhop/stats

  grep "Gx_CCR-" bulk-*.csv

• Response time of CCR-I/CCR-U/CCR-T messages at GW.

• Count of session in PCRF and count of session in GW. There could be some mismatch between the count due to time gap between determining session count from CPS and GW. If the count difference is high then it could indicate stale sessions on PCRF or GW.

• Count of AAR/RAR/STR/ASR messages from/to Application Function.

• Count of failed AAR/RAR/STR/ASR messages from/to Application Function. If Application Function has capability, capture details at error code level.

  Run the following command on pcrfclient01:

  cd /var/broadhop/stats

  grep "Gx_CCR-" bulk-*.csv

• Response time of AAR/RAR/STR/ASR messages at Application Function.

• Count of session in PCRF and count of session in Application Function. There could be some mismatch between the count due to time gap between determining session count from CPS and Application Function. If the count difference is high then it could indicate stale sessions on PCRF or Application Function.

  Count of session in PCRF:

  session_cache_ops.sh -count

Backup all relevant information to an offline resource. For more information on backup see Cisco Policy Suite Backup and Restore Guide.

• Data - Backup all database information. This includes Cisco MsBM Cisco Unified SuM.

Note	Sessions can be backed up as well.

• Configurations - Backup all configuration information. This includes SVN (from PCRF Client) the /etc/broadhop directory from all PCRFs
• Logs - Backup all logs for comparison to the upgrade. This is not required but will be helpful if there are any issues.

• Have proper sign off for any change request. Cisco and all customer teams must sign off.
• Make sure the proposed procedures are well defined.
• Make sure the rollback procedures are correct and available.

• Determine if the software upgrade will cause an outage and requires a maintenance window to perform the upgrade.
• Typically software upgrades can be done on one node a time and so minimize or eliminate any outage.
• Most of the time an upgrade requires a restart of the application. Most applications can be started in less than 1 minute.

• LINUX must be shutdown normally for VM restarts.
• All VMs are Linux.
• The preferred methods are init 0 or shutdown –h
• Failure to use the Linux OS shutdown can result in VM corruption and problems restarting the VM and applications.
• VM restart is typically done to increase resources to the VM (disk memory CPU).

• Hardware restarts should be rare.
• When a hardware restart is needed VMs must be shutdown first.
• When all VMs are stopped shutdown the hardware with either the ESXi console or as a power off.

• Planned outages are similar to hardware restarts.
• VMs need to be shutdown hardware can then be stopped.
• When hardware is started the typical hardware starting order is:

  • Start the servers with PCRFClient01 LB01 and SessionMgr01 first.

  • Start all other servers in any order after that.

To determine the disk space used use these Linux disk usage and disk free commands

• du
• df

The following details need to be captured for diameter issues:

• Details of service associated with subscribers in failure case.
• Pcaps capturing calls having issue.
• If the issue is with no response pcap should be captured both at CPS and the peer.
• Subscriber trace information can be captured using the following process

  • To add the subscriber that needs to be traced

    /var/qps/bin/control/trace_ids.sh -i <msisdn/imsi> -d sessionmgr01:<port no>/policy_trace

    cd /var/qps/bin/control

  • Run the following command to obtain subscriber information

    /var/qps/bin/control/trace.sh -i <msisdn/imsi> -d sessionmgr01<port no>/policy_trace

If CPS receives the request message for the same subscriber the trace result will be displayed.

Note	Port no. can be found in “Trace DB Database” configuration in Cluster-1. If Trace Database is not configured then by default “Admin Db Configuration” will pick up the trace database.

When you execute diagnostics.sh script on pcrfclient01 VM and it shows the following errors related to diameter proxy

For more information on diagnostics.sh, refer to diagnostics.sh section in CPS Operations Guide.

diameter_proxy-lb01_A DOWN L4CON 
Sessions (current,max,limit): 0,0, Rate (sessions,max,limit): 0,0, Last Status change (seconds): 2513094 
diameter_proxy-lb01_B DOWN L4CON 
Sessions (current,max,limit): 0,0, Rate (sessions,max,limit): 0,0, Last Status change (seconds): 2513093 
diameter_proxy-lb01_C DOWN L4CON 
Sessions (current,max,limit): 0,0, Rate (sessions,max,limit): 0,0, Last Status change (seconds): 2513092 
diameter_proxy-BACKEND DOWN 
Sessions (current,max,limit): 0,0,2000 Rate (sessions,max,limit): 0,0, Last Status change (seconds): 2513092 

The error L4CON message indicates that there is connection problem (e.g. “Connection refused” or “No route to host”) at layer 1-4. And the error message diameter_proxy-BACKEND DOWN signifies that all the service specified in diameter_proxy section in haproxy.cfg file are down.

1. Check whether HAProxy is running on load balancer VM. Specifically for this error message we should check in lb01.

2. Check the HAProxy configuration:

   vi /etc/haproxy/haproxy.cfg

   It should show similar entries as shown below. Try to telnet to corresponding load balancer VM with corresponding ports:

   diameter_proxy-lb01_A DOWN L4CON 
   Sessions (current,max,limit): 0,0, Rate (sessions,max,limit): 0,0, Last Status change (seconds): 2513094 
   diameter_proxy-lb01_B DOWN L4CON 
   Sessions (current,max,limit): 0,0, Rate (sessions,max,limit): 0,0, Last Status change (seconds): 2513093 
   diameter_proxy-lb01_C DOWN L4CON 
   Sessions (current,max,limit): 0,0, Rate (sessions,max,limit): 0,0, Last Status change (seconds): 2513092 
   diameter_proxy-BACKEND DOWN 
   Sessions (current,max,limit): 0,0,2000 Rate (sessions,max,limit): 0,0, Last Status change (seconds): 2513092 

If your Diameter Peer connectivity is down check the following:

1. Check the TCP connection on the diameter port (i.e.,) “netstat -pant | grep 3868". It should be in established state.

2. If the TCP connection is not getting established disable the firewall service iptables stop and check the port status /opt/broadhop/installer/support/add_open_port.sh pcrf 3868.

3. Open the Internet browser and go to your repository and check the published policies in runtime environment. You should notice the following configuration. If the following configuration is not there, then most probably it is a bad publish.

   DiameterConfiguration-_4davIF2KEeOXe-MDH-2FEQ.xmi

   DiameterStack-default-_A5cgQF2LEeOXe-MDH-2FEQ.xmi

4. If the problem is not in CPS and something is mis-configured in PCEF then you may notice the following messages in CPS

   tail -f /var/log/broadhop/service-qns-1.log

   Sending Alert Notification for host pcef realm lab.realm is down
   Sending Alert Notification for host pcef realm lab.realm is back up
   Sending Alert Notification for host pcef realm lab.realm is down
   Sending Alert Notification for host pcef realm lab.realm is back up

If the CPS server does not start (or starts and immediately crashes) and no errors appear in /var/log/broadhop/qns.log file to give reasons it did not start check the following list

1. Check /var/log/broadhop/service-qns-1.log file.

2. Check /etc/broadhop/servers.

   • There should be an entry in this file for the current host name (Type 'hostname' in the console window to find the local hostname)

   • There must be directory that corresponds to the hostname entry with config files. That is if the servers file has svn01=controlcenter there must be a /etc/broadhop/controlcenter directory.

3. Attempt to start the server directly from the command line and look for errors.

   • Type: /opt/broadhop/qns/bin/qns.sh

   • The server should start up successfully and the command line should not return. If the command prompt returns then the server did not start successfully.

   • Look for any errors displayed in the console output.

4. Look for OSGi errors.

   • Look in /opt/broadhop/qns/configuration for a log file. If any exist examine the log file for error messages.

A 401 type error means you're not logging in to SVN with proper credentials.

The server won't start and the following appears in the log:

2010-12-10 01:05:26,668 \[SpringOsgiExtenderThread-8\] 
ERROR c.b.runtime.impl.RuntimeLoader - There was an error 
initializing reference data\! 
java.io.IOException: Server returned HTTP response code: 
401 for URL: http://lbvip01/repos/run/config.properties 
sun.net.www.protocol.http.HttpURLConnection.getInputStream 
(HttpURLConnection.java:1313) \~\[na:1.6.0_20\] 
org.springframework.core.io.UrlResource.getInputStream(Url 
Resource.java:124) \~\[org.springframework.core_3.0.0.REL 

To fix this error:

• Edit /etc/broadhop/qns.conf

• Ensure that the configuration URL and repository credentials hostnames match.

  \-Dcom.broadhop.config.url=http://lbvip01/repos/run/

  \-Dcom.broadhop.repository.credentials=broadhop/

  broadhop@lbvip01

Symptoms server won't stay started and the log displays this:

com.broadhop.exception.BroadhopException: Unable to find system configuration for system: 
The system that is set up in your Quantum Policy Builder (and cluster name) must match the one  
specified in /etc/broadhop/qns.conf. Either add or change this via the Quantum Policy Builder  
interface, and then publish or update the system/clustername in /etc/broadhop/qns.conf 
\-Dcom.broadhop.run.systemId=poc-system 
\-Dcom.broadhop.run.clusterId=cluster-1 

For example:

<subscriberProfile><content/></subscriberProfile>

Because there are multiple ways needed to return web service data, the BroadHop Web Service Blueprint doesn't return any XML by default. To fix this issue, configure the 'Default Web Service Query Response' blueprint under the 'BroadHop Web Services' Blueprint.

com.broadhop.policy.impl.RulesPolicyService - Error 
processing policy request: Unknown action: 
com.broadhop.pop3auth.actions.IPOP3AuthRequest and Remote 
Actions are disabled. 

If you see an error of the type above, it means that the implementation class it's looking for is not available on the server. This can be caused by:

• The component needed is not installed on the server.
• Ensure that the pop3auth service is installed in your server.
• Look for exceptions in the logs when starting up.
• Try restarting the service bundle (pop3auth service in this case) using the OSGi console and looking at the logs.

You can check the configured NTP servers through # chronyc sources

In HA environment if we see some connection refused errors stop the firewall and execute

service iptables stop

to see if the problem is related to the iptables firewall issue.

2010-12-12 18:51:32,258 [pool-4-thread-1] ERROR 
c.b.licensing.impl.LicenseManager - Unknown error in 
logging 
java.lang.NullPointerException: null 
at 
com.broadhop.licensing.impl.LicenseManager.checkFeatures(L 
icenseManager.java:311) ~[na:na] 

This issue may occur if no license has been assigned yet.

Option 1: If this is for development or Proof Of Concept deployments you can turn on developer mode. This effectively gives you 100 users but is not for use in production.

1. Login to CPS.

2. Add the following to the /etc/broadhop/qns.conf file:

   -Dcom.broadhop.developer.mode=true

3. Restart CPS

Option 2: Generate a real license. Have your Cisco technical representative send you the Technical Article Tool com.broadhop.licensing.service - Creating a CPS License.

Option 3: If we have license error in the logs, check the MAC address of the VM and compare that with the MAC address in the license file in /etc/broadhop/license/.

This is likely caused because the server's name is not set up in the hosts file with its proper IP address.

In /etc/hosts the hostname (e.g. qns01) SHOULD NOT be aliased to 127.0.0.1 or localhost.

If improperly aliased JMX tells the server it's connecting to connect back with the IP of it's hostname. If it's aliased to localhost (127.0.0.1) the server attempts to open connections with itself which is unfortunate.

Example Error:

ERROR com.broadhop.management.JmxClient -

Unable to connect to JmxClient iomgr019045. Cause no

such object in table Will attempt to reconnect.

During machine boot fsck is run on file systems to check its consistency. This consistency check is done without user intervention and automatically fixes errors which it can. But sometimes if there is a hard reset to CPS VM/machine for example because of abrupt power failure then during fsck all the problems are not automatically fixed and user intervention is must to fix the errors reported by fsck. The table below describes the common fsck errors along with their description and solution.

• The link gives list of all the errors which are automatically fixed by fsck as well as list of errors where user intervention is must -

• The link gives general idea about various phases in fsck.

• The link describes all the errors in case of UFS file system.

This link can be used as a reference to fix the errors reported by fsck on CPS file system which is ext3.

CPS memory consumption can be monitored using appropriate KPIs in Grafana graphs or other monitoring tools. If memory consumption increases beyond the default threshold of 90% on any CPS VM CPS will generate a Low Memory alarm for that VM. This threshold is configurable in the CPS Deployment Template using the free_mem_per setting.

Debit compression can be used to identify what all the debits have happened for the subscriber. This data can also be used to cross check the debits with external entities.

• To disable compression add/edit the following flag in /etc/broadhop/qns.conf file.

  -DcompressDebits=false

• To enable compression add/edit the following flag in /etc/broadhop/qns.conf file.

  -DcompressDebits=true

We can also check directly in mongo how balance has been debited /credited for subscriber using the following queries

{ 
        “_id” : ObjectId(“001000009576290454afdc77”), 
        “_id_key” : “001000009576290454afdc77”, 
        “name_key” : { 
 
        }, 
        “end_date_key” : null, 
        “realm_key” : null, 
        “parent_id_key” : null, 
        “billing_info_key” : { 
                “rate_plan_code_key” : null, 
                “charging_id_key” : null 
        }, 
        “status_key” : "ACTIVE", 
        “version_key” : 0, 
        “start_date_key” : null, 
        “credentials_key” : [ 
                { 
                        “network_id_key” : "111", 
                        “description_key” : null, 
                        “password_key” : null, 
                        “type_key” : null, 
                        “expiration_date_key” : null 
                } 
        ], 
        “role_key” : “READ_ALL”, 
        “external_id_key” : null, 
        “_transId” : “d2a3f602-69bb-4047-af6f-c979ec36732f-1” 
} 

• Check whether you are getting any errors in diagnostics.sh and try to fix the error.

• Make sure that you have the correct URL for the run time environment. For example http//pcrfclient01/repos/run

• Make sure that you have following configuration on /etc/broadhop/pb/pb.conf configuration file. If not then do the changes as shown below and run the synconfig.sh and restartall.sh for the changes to come into effect:

  Caution

  Executing restartall.sh will cause messages to be dropped.

  Sample Configuration:

  SESSION_TIMEOUT="-Dsession.timeout=9000" 
  QNS_SESSION_DATABASE="-Dsession.db.primary=sessionmgr01 -Dsession.db.secondary=sessionmgr02 -Dsession.db.port=27717 
  -Dua.client.submit.audit=false" 
  HA System Sample Configuration: 
  SESSION_TIMEOUT="-Dsession.timeout=9000" 
  QNS_SESSION_DATABASE="-Dsession.db.primary=sessionmgr01 -Dsession.db.secondary=sessionmgr02 -Dsession.db.port=27717 
  -Dua.client.submit.audit=true 
  -Dua.client.server.url=http://:8080 

  Note	The IP-address is usually LBVIP01 where the SOAP requests are sent to Unified API for our configuration.

• If you still face issue collect and analyze the following logs:

  • /var/log/broadhop/qns-engine-pb.log

  • /var/log/broadhop/service-qns-pb.log

    .

• Check whether the “snmpd” process is running in respective VM with the command monit status snmpd. If it is stopped start the snmpd process with the command monit start snmpd.

• Check whether all the IP tables have been turned off and check the status of UDP port 162 provided you are using the same UDP port 162 at the NMS as well.

• Check the external NMS IP is defined in policy director (lb) VM under /etc/hosts and also in the /etc/snmp/scripts/component_trap_convert in place of corporate_nms_ip.

• Check the file cat /etc/snmp/snmpd.conf has the line “rocommunity Broadhop” because all the internal traps from various policy server (QNS) VM to active policy director (lb) VM is been sent over this default community name “Broadhop” as mentioned above.

• Check the trap community name is same both in policy director and as well as in external NMS system. For example, cat /etc/snmp/scripts/snmp_communities trap_community=cisco (customer external NMS system should also have this same “cisco” community name.

• Check whether the traps from respective policy server (QNS) VM is properly reaching active policy director (lb) VM this can be checked under /var/log/snmp/trap.

• Check for /var/log/messages on active policy director (lb) for further analysis.

When an hapoxy load balancer which forwards request to Policy Builder server on pcrfclient01 is not available then it forwards the request to backup server on pcrfclient02.

Consider pcrfclient01 is up and a new repository is added using Policy Builder. This repository is saved on pcrfclient01 (on file at /etc/broadhop/pb/policyRepositories.xml /etc/broadhop/pb/publishRepositories.xml).

If pcrfclient01 becomes inaccessible haproxy sends request to pcrfclient02 where it does not find the above mentioned two files (publishRepositories.xml policyRepositories.xml) and does not display any repository on PB GUI.

Make sure the IPv6 firewall is disabled on lb01and lb02. If the firewall is not disabled then you can disable it by executing the command

service ip6tables stop

ZeroMQ connection established between Policy Director (lb) and other site Policy Server (qns).

L2-CA-SEC-lb01  2015-05-10 18:58:04,943 [pool-2-thread-1] ERROR c.b.d.impl.server.StackManager - Stack 
is Null and Realm is not found null:16777238 -  realmToStacks [ocs1.sy.server.cisco.com:7,  
ocs3.sy.server.cisco.com:7, ocs4.sy.server.cisco.com:7, cscf3.cisco.com:16777236,  
ocs2.sy.server.cisco.com:7, cscf6.cisco.com:16777236, ocs6.sy.server.cisco.com:7,  
ocs5.sy.server.cisco.com:7] 
L2-CA-SEC-lb01  2015-05-10 18:58:04,944 [pool-3-thread-1] WARN  c.b.d.impl.server.StackManager -  
Dropping message Outbound: Cmd: 272/1/0 E2E: 1431976189, HBH: 2798797074, Session-ID:  
ds4;333241;2883160674, Result-Code: 2001 
L2-CA-SEC-lb01  2015-05-10 18:58:04,944 [pool-2-thread-1] ERROR c.b.d.impl.server.StackManager - Stack  
is Null and Realm is not found null:16777238 -  realmToStacks [ocs1.sy.server.cisco.com:7,  
ocs3.sy.server.cisco.com:7, ocs4.sy.server.cisco.com:7, cscf3.cisco.com:16777236,  
ocs2.sy.server.cisco.com:7, cscf6.cisco.com:16777236, ocs6.sy.server.cisco.com:7,  
ocs5.sy.server.cisco.com:7] 
L2-CA-SEC-lb01  2015-05-10 18:58:04,944 [pool-3-thread-1] WARN  c.b.d.impl.server.StackManager -  
Dropping message Outbound: Cmd: 272/2/1 E2E: 1431980725, HBH: 2798442695, Session-ID:  
ds1;333241;2799910481, Result-Code: 2001 

If Upgrade from Existing 7.0 System does not show latest version then perform the following steps to show the latest version:

1. Reinitialize your environment by executing the following command from Cluster Manager:

   /var/qps/install/current/scripts/upgrade/reinit.sh

2. To restart all the policy server (qns) services execute the following command from Cluster Manager:

   /var/qps/install/current/scripts/bin/control/restartall.sh

   Caution

   Executing restartall.sh will cause messages to be dropped.

3. Verify CPS Status by running diagnostics.sh and about.sh scripts from Cluster Manager.

   For more information on diagnostics.sh, refer to diagnostics.sh section in CPS Operations Guide.

Scenario 1: When the svn-repos password expires the Policy Builder opens only in Read-only mode.

Scenario 2: Invalid username or password.

Note	For more information about this and other CPS administrative commands, refer to the CPS Operations Guide.

Case: Graphs in Grafana are lost when system time on VMs are changed.

Solution: Change the system timing on all VMs. Also change the browser time according to graphite server time and restart the collectd service on each VM.

The graphite server time and browser time should match then only we will be able to see graphs.

Case: Systems configuration is not displayed for Plugin Configuration in Reference Data tab.

Possible Cause: This issue could occur if Systems plugin configuration is configured using system.json file.

Solution: Check whether your system.json file is valid or not using any json validator.

Case: Publishing is not available

Possible Cause: SVN configuration is manually exported and imported from one setup to another. While performing import user missed to import .broadhopFileRepository or deleted it unknowingly.

Solution: Check whether .broadhopFileRepository is present in pcrfclient01. If it not present import the file.

Problem: There is an impact on 7.5.0 and higher releases with a new feature “check to switch to unknown service if subscriber is deleted mid session” due to the custom policies defined in some customer locations.

Solution: For the customers who are on pre-7.5.0 release and don't want the new feature a work around has been suggested with an addition of custom policy that will bypass this feature.

The custom policy has to be added in the call flow based on the conditions. This custom policy will add a “IgnoreSPR” AVP to policy state. If this AVP is present internally the code will skip the feature.

Below are screenshots of the policy where the customer wants to skip the feature in some specific conditions of flow like when “Authenticating a subscriber on AAA server” since we don't store subscriber information in SPR.

Without this custom polices we will see session switching from known to unknown service during SPR update/Accounting. This will be visible in information logs in engine and policy server (qns) with “Session has switched from known to unknown as subscriber could not be found”.

Conditions

Actions

Issue: Policy Builder is not able to build indexes for table (Custom Reference Table).

Case: While publishing Policy Builder CPS logs below exception in policy server (qns) log.

For example,

ERROR c.b.custrefdata.impl.dao.GenericDao - Could not build indexes for table

QoS-Reference-Mapping

com.mongodb.CommandFailureException

Possible Cause: This could happen when CRD table key columns are changed from back-end (xmi) in Policy. Due to this underlying composite index on CRD table does not reflect new/changed key columns.

Solution: Drop the index on CRD table in MongoDB and publish the Policy Builder.

1. Drop index manually.

   db.<crdtablename>.dropIndexes()

2. Make sure xmi (backend) and Policy Builder data of CRD table whose index you want to drop are in sync.

   If both are not in sync, CPS displays There are uncommitted changes to the '<PBrepositoryname>' repository. Do you wish to discard those changes? while logging to the Policy Builder.

   For example, if CRD table data gets modified via backend (xmi) then when you login, CPS shows uncommitted message. Choosing Retain will sync up the xmi and Policy Builder.

3. Publish Policy Builder.

4. Check the rebuilt index.

   db.<crdtablename>.getindexes()

Case: Messages timed out intermittently. CPS logs reports following exceptions

2015-10-11 145054918 [pool-2-thread-1] ERROR c.b.d.p.event.DiameterMessageDealer.? - Error

submitting message to lb

2015-10-11 145054918 [pool-2-thread-1] ERROR c.b.d.p.event.DiameterMessageDealer.? - Error

submitting message to lb

Possible Cause: Message timed out intermittently problem happens when a GC pause greater than 10 seconds is occurring on policy server (qns) and policy director (lb). Due to this pause queue gets overloaded and there are message drops and timeouts. This pause happens when the service-qns logs are getting rotated with size 100 M.

Solution: The following changes need to be done on cluster manager

• Change Daily > hourly, size 100M > 25M and rotate 5 > 20

  cat /etc/logrotate.d/qps 
  /var/log/broadhop/determine_cluster_state.log 
  /var/log/broadhop/service-qns-*.log 
  /var/log/elasticsearch/*.log 
  { 
       daily 
       nodateext 
       copytruncate 
       size 25M 
       rotate 20 
       missingok 
       compress 
  } 

• Copy the changes to all the VMs using copytoall command.

Case: There are no sessions on CPS but the statistics count still showing statistics.

#session_cache_ops.sh --count 
Session cache operation script 
Fri Nov 13 01:26:08 EST 2015 
------------------------------------------------------ 
Session Replica-set SESSION-SET1 
------------------------------------------------------ 
Session Database          : Session Count 
------------------------------------------------------ 
 session_cache            : 0 
 session_cache_2          : 0 
 session_cache_3          : 0 
 session_cache_4          : 0 
------------------------------------------------------ 
 No of Sessions in SET1   : 0 
------------------------------------------------------ 
 
Total Number of Sessions  : 0 
 
#session_cache_ops.sh --statistics-count 
Session cache operation script 
Fri Nov 13 01:26:31 EST 2015 
------------------------------------------------------ 
 Sessions statistic counter on Genaral 
------------------------------------------------------ 
  Session Type         : Session Count 
------------------------------------------------------ 
ADMIN-SET1 
  RX_TGPP              : 364 
  GX_TGPP              : 983269 
  SY_PRIME             : 974457 
------------------------------------------------------ 
# 

Possible Cause: CPS monitors the session count and updates the aggregation of message type into counters collection in the admin database. This query is performed on secondary databases. If due to some reason all secondary members are not in healthy state or are in recovering state, then we can incur that the discrepancy is in session count.

mongo rtpclabqps5g-sm01a:47721 
MongoDB shell version: 2.6.3 
set05:PRIMARY> use sharding 
set05:PRIMARY> db.counters.find() 
{ "_id" : 8, "db" : "session_cache_3", "session_type" : [ ] } 
{ "_id" : 9, "db" : "session_cache_4", "session_type" : [ ] } 
{ "_id" : 10, "db" : "session_cache", "session_type" : [ { "type" : "SY_PRIME", "count" : 246563 },  
{ "type" : "GX_TGPP", "count" : 248921 }, { "type" : "RX_TGPP", "count" : 93 } ] } 
{ "_id" : 11, "db" : "session_cache_2", "session_type" : [ { "type" : "SY_PRIME", "count" : 247330 },  
{ "type" : "GX_TGPP", "count" : 249614 }, { "type" : "RX_TGPP", "count" : 94 } ] } 
{ "_id" : 12, "db" : "session_cache_3", "session_type" : [ { "type" : "SY_PRIME", "count" : 227624 },  
{ "type" : "GX_TGPP", "count" : 229542 }, { "type" : "RX_TGPP", "count" : 90 } ] } 
{ "_id" : 13, "db" : "session_cache_4", "session_type" : [ { "type" : "SY_PRIME", "count" : 252940 },  
{ "type" : "GX_TGPP", "count" : 255192 }, { "type" : "RX_TGPP", "count" : 87 } ] } 
{ "_id" : 18, "db" : "session_cache_2", "session_type" : [ ] } 

Diagnostic showing all secondary members are in bad shape:

rtpclabqps5g-qns09b  rtpclabqps5g-qns09b 2015-11-13 03:06:45,603 [pool-2-thread-1] WARN   
c.b.c.m.dao.impl.ShardInterface - Unable to get direct connection for DB shard { "_id" : 10 ,  
"seed_1" : "sessionmgr21" , "seed_2" : "sessionmgr22" , "port" : 27737 , "db" : "session_cache" ,  
"online" : true , "count" : 0 , "backup_db" : false , "lockTime" : { "$date" :  
"2015-11-13T08:06:25.997Z"} , "isLocked" : false , "lockedBy" :  null } - bypassing type counts 
rtpclabqps5g-qns09b  rtpclabqps5g-qns09b 2015-11-13 03:06:45,605 [pool-2-thread-1] WARN   
c.b.c.m.dao.impl.ShardInterface - Unable to get direct connection for DB shard { "_id" : 11 ,  
"seed_1" : "sessionmgr21" , "seed_2" : "sessionmgr22" , "port" : 27737 , "db" : "session_cache_2" ,  
"online" : true , "count" : 0 , "backup_db" : false 

Case: After CPS upgrade disk statistics are not populated in Grafana.

Possible Cause: Configurations are not refreshed after collectd package is upgraded.

Solution: Restart collectd service on respective VM/VMs.

Case: On running a load with Total TPS of 1200 for a CPS having four policy servers (qns) and for 500000 subscribers it was seen that for some of the CCR-U request the CPS sends “Session has switched from known to unknown as subscriber could not be found” causing the CCA-U to give a result code of 5012.

The call model that is used here is a simple Gx call model involving several CCR-U for Charging-Rule-Report were the subscribers are provisioned in a Auto-provisioning manner.

Possible Cause: The call model being run was auto provisioning call model with 200 TPS CCR-I 800 CCR-U and 200 CCR-T. On every CCR-I we had a subscriber being automatically provisioned and the balance provisioned automatically with the Automatic Balance Provisioning service.

We saw lot of locking errors for balance being the cause as there was a version mismatch being seen while updating balance.

qns02 qns02 2016-02-11 06:11:21,231 [pool-1315-thread-1] ERROR c.b.b.i.d.i.MongoBalanceRepository -  
Cache data is out of date for object 0057170054ce8d1a56bc6c59 
qns03 qns03 2016-02-11 06:11:22,041 [pool-1308-thread-1] WARN  
c.b.b.i.a.AutowireBalanceManagerBlueprint - Couldn't find a current Account Balance Status for  
template: daily 
qns02 qns02 2016-02-11 06:11:21,232 [pool-1315-thread-1] WARN c.b.d.p.g.t.DiameterGxTGPPDeviceMgr -  
Issue getting reservation status for external reservation id: 1234ds1;338812;2613626736, Balance  
Code 1234, Subscriber Id: 0057170054ce8d1a56bc6c59 
com.broadhop.exception.CachedDataIsOutOfDate: Optimistic Locking Error - the version number does  
not match the database version for subscriber: 0057170054ce8d1a56bc6c59 

Solution: Thus with balance auto provisioning enabled and high TPS of balance provisioning (high CCR-I TPS which causes balance to be provisioned) it is suggested to keep the Db Read Preference as Primary or PrimaryPreferred under Balance Configuration plug-in in Policy Builder. This will avoid the balance locking errors.

Case: Sometimes the RAR message is not sent out from policy director (lb) even though CPS records in engine logs that the message (RAR, ASR and so on) has been sent. It is an intermittent behavior.

The following logs can be seen on the occurrence of this issue:

qns02  qns02 2016-02-22 18:07:31,634 [pool-2-thread-1] DEBUG c.b.d.p.registry.EndpointRegistry - No  
endpoint available and current endpoint is down lb01-4:diameter-lb site null 
qns02  qns02 2016-02-22 18:07:31,634 [pool-2-thread-1] DEBUG c.b.d.p.registry.EndpointRegistry -  
Unable to get alternate endpoint for realm site-rx-client.com, host site-host-rx. Setting  
destination to null. 

Possible Cause: This issue may occur if the correct value of the parameter -Ddiameter.peer.reload.interval is not configured in /etc/broadhop/qns.conf file.

CPS reloads the peer endpoints after every 30 seconds. It also reloads endpoints whenever there is an occurrence of peer flapping or new peer tries to connect.

To avoid unnecessary reloading of endpoints CPS checks that if endpoint reload request comes within the 3 second interval after 30 seconds regular reload. If the reload request does not come within the stipulated time CPS does not allow to reload.

Sometimes if request comes within this 3 second interval then the request is not processed and endpoints are not loaded due to which the message in question at that time will not be sent out from policy director (lb) though it is visible in engine logs that the message has been sent out.

Solution: This 3 second interval can be tweaked using -Ddiameter.peer.reload.interval parameter in /etc/broadhop/qns.conf file.

If it is kept to default value (0 millisecond) then there is a very less probability of collision so a 0 millisecond or very small value is advisable.

Case: Sometimes the time zone information and the location information are not received in AAA response message from CPS.

Case: Admin database shows problem in connecting to the server in diagnostics. It throws the following error message while checking replica set status.

diagnostics.sh --get_replica_status

Current setup have problem while connecting to the server on port 27721

Possible Cause: The oplog collection is a circular capped collection. It is possible that the corruption occurred due to abrupt failure of VM and exception comes when the collection wrapped around to the corrupted region.

• Check which specific replica-set member is corrupted. It can be verified using rs.status() command.

  For example,

  mongo sessionmgr01:27721 
  >rs.status() 
  "_id" : 3, 
  "name" : "L3-CA-SEC-sessionmgr01:27721", 
  "health" : 1, 
  "state" : 2, 
  "stateStr" : "SECONDARY", 
  "lastHeartbeatMessage" : "syncThread: 17322 write to oplog failed: InternalError no space in capped collection", 
  "syncingTo" : "L3-CA-SEC-sessionmgr02:27721" 

• Also verify if mongo logs shows the following related errors:

  2016-03-09T14:33:24.101+0530 [rsHealthPoll] replSet member L3-CA-SEC-sessionmgr01:27721 is up 
  2016-03-09T14:33:24.101+0530 [rsHealthPoll] replSet member L3-CA-SEC-sessionmgr01:27721 is now in state SECONDARY 
  2016-03-09T14:33:29.801+0530 [rsSync] couldn't make room for new record (len: 172) in capped ns local.oplog.rs 
  2016-03-09T14:33:29.801+0530 [rsSync]   Extent 0 
  2016-03-09T14:33:29.801+0530 [rsSync]  (capExtent) 
  2016-03-09T14:33:29.801+0530 [rsSync] 
  2016-03-09T14:33:29.801+0530 [rsSync]     magic: 41424344 extent->ns: local.oplog.rs 
  2016-03-09T14:33:29.801+0530 [rsSync]     fr: null lr: 1:1b8dedd4 extent->len: 1073741824 
  2016-03-09T14:33:29.801+0530 [rsSync] local.oplog.rs Assertion failure len * 5 > _lastExtentSize  
  src/mongo/db/structure/catalog/namespace_details.cpp 366 

Solution: Make sure there is at least one surviving member that is primary database member using rs.status() command.

1. Stop mongo process.

   /etc/init.d/sessionmgr-27721 stop

2. Go to the data directory.

   For example,

   cd /var/data/sessions.3

3. Take backup of local file at a temporary location.

   ls -l local*

   -rw------- 1 root root 67108864 Jan 7 2253 local.0

   -rw------- 1 root root 2146435072 Jan 27 0251 local.1

   -rw------- 1 root root 16777216 Jan 27 0251 local.ns

4. Remove the local files.

   rm -rf local.*

5. Start the mongo process.

   /etc/init.d/sessionmgr-27719 start

6. Check whether the local files have been re-created again.

   ls -l local*

   -rw------- 1 root root 67108864 Jan 7 2253 local.0

   -rw------- 1 root root 2146435072 Jan 27 0251 local.1

   -rw------- 1 root root 16777216 Jan 27 0251 local.ns

7. Repeat Step 1 to Step 6 for other corrupted member.

Case: Locale environment variables related errors are observed by MAC users while running CPS scripts/logging into CPS VMs, and so on.

Possible Cause: CPS VMs show -bash: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory warning while login through SSH and if we login to such terminal then there is some issue while creating replica-sets.

Solution:

1. Go to Terminal > Preferences > Profiles > Advancedtab.

   
   

   

   
   

2. Uncheck Set locale environment variables on startup and restart your terminal.

Issue: If the user has recently added/modified shards but is seeing all the sessions created in first shard only.

Symptoms: Run the session_cache_ops.sh script and check the session count. Here in this case all sessions are stored in shard 1 (102543) as the rest of the shards have count 0.

Session cache operation script
Thu Jun 16 02:07:45 EDT 2016
------------------------------------------------------
Session Replica-set SESSION-SET1
------------------------------------------------------
Session Database          : Session Count
------------------------------------------------------
 session_cache            : 102543
 session_cache_2          : 0
 session_cache_3          : 0
 session_cache_4          : 0
------------------------------------------------------
 No of Sessions in SET1   : 0
------------------------------------------------------

------------------------------------------------------
Session Replica-set SESSION-SET2
------------------------------------------------------
Session Database          : Session Count
------------------------------------------------------
 session_cache            : 0
 session_cache_2          : 0
 session_cache_3          : 0
 session_cache_4          : 0
------------------------------------------------------
 No of Sessions in SET2   : 0
------------------------------------------------------

Total Number of Sessions  : 0

Issue: Licensing is not throwing traps or diagnostic errors upon breach.

Symptoms: Application traps are not generated.

Solution: Check log level for logger com.broadhop.eventlogging in /etc/broadhop/qns.conf file.

<logger name="com.broadhop.eventlogging" level="info">
    <appender-ref ref="JSON-LOGGER" />
  </logger>

Important

Log level must be set to info.

If log level is set to warn/error, SNMP traps related to licensing will not be generated. You need to change the log level to info to generate traps related to licensing.

After changing the log level to info, execute the following commands:

/var/qps/install/current/scripts/build_all.sh

/var/qps/install/current/scripts/upgrade/reinit.sh

Issue: When firewall state is changed from enabled to disabled state or vice versa, sometimes firewall is not completely purged on some VMs.

Issue: Too many request to query mongo.

Case: Consider user is running a very basic call model (Gx only) on CPS setup. On reaching close to 15K TPS (1 CCR-I, 3- CCR-U and 1-CCR-T), timeouts are observed in grafana and average response time goes up.

[root@pcrfclient01 ~]# mongostat --host sessionmgr01 --port 27717 
connected to: sessionmgr01:27717 
insert query update delete getmore command flushes mapped vsize res faults locked db idx miss % qr|qw ar|aw netIn 
netOut conn set repl time
1055 17691 3868 921 248 328|0 0 5.03g 5.84g 4.12g 0 session_cache:32.0% 0 73|0 0|4 19m 33m 137 set01 PRI 15:26:15
973 16848 3568 752 200 257|0 0 5.03g 5.84g 4.12g 0 session_cache_2:29.7% 0 4|0 6|1 18m 31m 138 set01 PRI 15:26:16
1049 17023 4162 809 215 283|0 0 5.03g 5.84g 4.12g 0 session_cache_3:33.7% 0 74|1 2|3 20m 35m 138 set01 PRI 15:26:17
1010 17200 3956 804 211 278|0 0 5.03g 5.84g 4.12g 0 session_cache_4:32.1% 0 18|0 3|1 19m 33m 137 set01 PRI 15:26:18
975 17027 3912 816 206 275|0 0 5.03g 5.84g 4.13g 0 session_cache:31.8% 0 34|0 0|4 19m 33m 138 set01 PRI 15:26:19
990 16631 3643 904 204 279|0 0 5.03g 5.84g 4.13g 0 session_cache_2:34.4% 0 58|0 0|4 18m 31m 138 set01 PRI 15:26:20
913 15475 3806 881 185 257|0 0 5.03g 5.84g 4.13g 0 session_cache_4:30.7% 0 40|0 1|3 18m 32m 140 set01 PRI 15:26:21

If VIP name is modified then user has to manually delete old VIP from active policy director (lb)/OAM (pcrfclient) using the following below command:

pcs resource delete <old-vip-name>

where, <old-vip-name> is the old VIP name.

Issue: lbvip does not move cleanly to the secondary policy director (lb) VM when the network on primary policy director (lb) VM is stopped.

Scenario: For example, consider lbvip is on lb01 VM.

To stop the network on lb01 VM, execute the following command:

service network stop

lbvip moves to lb02 VM immediately but it is not pingable from anywhere which stops the traffic and grafana.

After performing service network restart on lb02 VM, the traffic restored partially with lot of errors (and lbvip is pingable from everwhere).

After stopping the network on lb01 VM, lbvip was seen on both the lb VMs (even after doing network restrat on lb02 VM).

Solution:

Before executing service network stop, stop corosyn from the node using monit stop corosync command.

Note	This is needed since corosync has the functionality to bring up an interface if they are down. So after service network stop is executed all interfaces are down and corosync brings up the interfaces (like, eth0:0, eth1:0, and so on).

Case: session_cache_ops.sh --count shows the error: There is no session db found.

Symptoms: On power outage, if all the members of session database replica were down and came up, session database replica-set will get automatically re-created. However after this if you run session_cache_ops.sh --count script, it may show the error: There is no session db found.

This is because, script do not create session_cache databases on its own. session_cache databases would get created by application automatically when calls would run.

Here is an example:

Session cache operation script
Mon Oct 17 05:40:34 EDT 2016

There is no session db found for site SITE1

There is no session db found for site SITE1

There is no session db found for site SITE1

There is no session db found for site SITE1

Validate: Run listshards command to verify the internal shards are not deleted.

For HA, run the following command:

echo "listshards" | nc qns01 9091

For Active active GR, run the following command:

echo "listshards <site name>" | nc qns01 9091

This command displays all the shards configured in the system.

Here is an example:

echo "listshards clusterA_PRI" | nc qns01 9091
osgi> 
Shard Id    Mongo DB                              State     Backup DB    Removed    Session Count

1           sessionmgr01:27717/session_cache      online    false        false      261758       
2           sessionmgr01:27717/session_cache_2    online    false        false      261439       
3           sessionmgr01:27717/session_cache_3    online    false        false      261139       
4           sessionmgr01:27717/session_cache_4    online    false        false      262069       
5           sessionmgr03:27722/session_cache      online    false        false      261984       
6           sessionmgr03:27722/session_cache_2    online    false        false      260759       
7           sessionmgr03:27722/session_cache_3    online    false        false      262147       
8           sessionmgr03:27722/session_cache_4    online    false        false      262087       
9           sessionmgr05:27723/session_cache      online    false        false      261627       
10          sessionmgr05:27723/session_cache_2    online    false        false      262118       
11          sessionmgr05:27723/session_cache_3    online    false        false      262088       
12          sessionmgr05:27723/session_cache_4    online    false        false      261775       
13          sessionmgr09:47717/session_cache      online    true         false      0            
14          sessionmgr09:47717/session_cache_2    online    true         false      0            
15          sessionmgr09:47717/session_cache_3    online    true         false      0            
16          sessionmgr09:47717/session_cache_4    online    true         false      0            

Rebalance Status: Rebalanced

After the ISSM or ISSM rollback, if some of the members are in the Unknown or Recovering state, then perform the following steps:

• Identify the problematic Replica-sets from diagnostics.sh.

• Perform the following steps in all the problematic members in Replica-sets in parallel.

  
  ssh sessionmgrxx "monit stop aido_cleint"
  ssh sessionmgrxx "/etc/init.d/sessionmgr-xxxxx stop"
  ssh sessionmgrxx "rm -rf data_path_folder_of_member"
  ssh sessionmgrxx "/etc/init.d/sessionmgr-xxxxx stop"
  ssh sessionmgrxx "monit start aido_cleint"

• Repeat the Step 2, for all problematic replica sets.

If the Flow-Information parameters are not derived as per the Actions (Enforce/Mirror), check for the following configuration details:

• Ensure that the AVP name defined in the ColumnAndAvpPair in the RxSTGConfiguration service option is the correct action name ("Flow Status" or "Flow Description) to derive the action. Also check the 3GPP AVP name to derive the CRD value for the field.

• Confirm that the action values in the CRD are correctly entered (Mirror/Enforce).

• Check if the Flow-Description value defined in the CRD for Enforce action is in correct format. If the IPFilterRule syntax is not proper then CPS logs a warn message to indicate the same.

• Enable logs at debug level and confirm the CRD evaluation and logs for applying the action of the individual fields in the flow information.

If the source AVP is a Session/Policy State Data Retriever, the target AVPs are sent in the outbound diameter messages only if the data to be retrieved is available in the Session/Policy State data.

Issue: After applying patch or updating kernel in HA setup, when you run puppet apply command /etc/httpd/conf/httpd.conf file was modified, not all VMs are configured with the modified httpd.conf file:

Solution: After applying a patch or updating kernel in HA setup, run the following command from Cluster Manager:

puppet apply --logdest=/var/log/cluman/puppet-custom-run.log -- modulepath=/opt/cluman/puppet/modules --config=/opt/cluman/puppet/ puppet.conf /opt/cluman/puppet/nodes/node_repo.pp

Note	Manually enter puppet apply command in your system.

After applying the puppet apply command, run the following command from Cluster Manager to update the /etc/httpd/conf/httpd.conf file on all VMs:

/var/qps/install/current/scripts/modules/update_httpd_conf.py

Case: Unable to complete rebalance operation displaying following error:

osgi> rebalance GR-S1
Rebalancing ...
Unable to complete operation.

Reason:

• Instance site1-qns90-1 is on older version (1) of sharding configuration. Latest version is: (10)

• Make sure all the instances are up and running and try running rebalance command again

• If the instance in the error does not exist, manually clear it from instances collection (admindb/sharding) and try running rebalance command again

Possible Cause: qns90 VM is either deleted or is no more in use or policy server (QNS) process is down

Solution:

If QNS process was down unintentionally, perform the following steps:

1. Start the policy server (qns) process.

2. Verify that the diagnostics is clear.

3. Run the rebalance command again.

If qns90 VM is deleted or is no more in use, perform the follow steps:

1. Login to the admin database.

2. Clear the instance in error from "instances" collection.

   use sharding

   1. Delete the instance in error.

      db.instances.find({"_id":"Site1-qns90-1"})

      db.instances.remove({"_id":"Site1-qns90-1"})

   2. Reload configuration.

      db.config.update({},{$inc:{"version": NumberInt(1)}})

      db.changes.insert({"ts":new Timestamp(), "change" : "Manually deleting qns90"})

3. Run the rebalance command again.

Case: pcrfclient01 automatically becomes unresponsive after some time in OpenStack environment

Condition: You are not able to SSH to any VM.

Solution: Refer to the following fix:

The Errata for OSP10 (Newton) is:

RHSA-2018*0058 - Security Advisory Issued:2018-01-05 Updated: 2018-01-05 Errata:

https://access.redhat.com/errata/RHSA-2018T0058

Fixes:

• BZ - 1519780 - CVE-2017-5715 hw: cpu: speculative execution branch target injection

• BZ - 1525502 - QEMU's subsystem gets stuck inhibiting all I/O operations on virtio-blk-pci devices [OSP 10]

Issue: If the primary database member gets isolated from all the arbiters then diagnostics output displays incorrect state for the primary member.

Solution: If a member is displayed in an unknown state when using diagnostics.sh --get_replica_status command, it is likely that the member is not accessible from one of other members, mostly an arbiter. In that case, you must go to that member and check its connectivity with other members.

Also, you can login to mongo on that member and check its actual status.

Case: No SNMP alarm is generated when Mongo process is stopped or restarted.

Possible Cause: This can happen due to the timing difference between AIDO functionality and SNMP alarm schedule.

• When the Mongo process is stopped, it gets restarted by AIDO.

• Before SNMP scheduler detects the Mongo failure, it is restarted by AIDO. Hence, no alarm is generated.

Solution:

If you want to stop AIDO from restarting the Mongo process, you need to create /var/tmp/stopped-PORT file on the AIDO client node.

This stops AIDO from monitoring or handling the Mongo process. The alarm is generated when Mongo process is stopped or restarted.

Example: During ISSU, you need to stop and start mongod process which is done by upgrade script. So, you need to execute the following steps:

1. Create /vat/tmp/stopped-PORT file on AIDO client node.

2. Stop mongod using /etc/init.d/sessionmgr-PORT stop

3. Upgrade your system (ISSU). AIDO does not restart the mongod process as /vat/tmp/stopped-PORT file is created on AIDO client node.

4. Start mongod using /etc/init.d/sessionmgr-PORT start

5. Remove /vat/tmp/stopped-PORT file from AIDO client node

Case: Zookeeper in an endless restart cycle. In monit summary, the zookeeper-server is displayed in Initiaizing state.

Process 'zookeeper-server'          Initializing

Possible Cause: Zookeeper server with version 3.4.6 fails to start.

Solution: Go to /var/opt/zookeeper/version-2 and delete zero byte file or move the file to some other location.

ls -l
total 1988
-rw-r--r-- 1 root root 67108880 Feb 20  2018 log.1
-rw-r--r-- 1 root root 67108880 Aug 23 04:58 log.1f43
-rw-r--r-- 1 root root        0 Aug 23 04:58 log.31fd

After moving out zero byte file from /var/opt/zookeeper/version-2, check monit summary. The zookeeper-server must be in running state.

Case: Upgrade fails when running puppet.

During upgrade, the following error is encountered:

2018-12-10 05:10:17,909 INFO [__main__.run_recipe] Performing installation stage: ApplyClumanPuppet
2018-12-10 05:10:17,910 INFO [cluman_puppet.run] Applying puppet on cluman
2018-12-10 05:10:53,133 ERROR [cluman_puppet.run] Puppet failed! For details check log /var/log/cluman/puppet-run.log
2018-12-10 05:10:53,133 ERROR [__main__.<module>] Error during installation
2018-12-10 05:10:53,133 ERROR [__main__.<module>]
Traceback (most recent call last):
  File "/mnt/iso/modules/install/__main__.py", line 780, in <module>
    main(sys.argv[1:])
  File "/mnt/iso/modules/install/__main__.py", line 770, in main
    install(argv[1:])
  File "/mnt/iso/modules/install/__main__.py", line 208, in install
    run_recipe(install_recipe)
  File "/mnt/iso/modules/install/__main__.py", line 763, in run_recipe
    stage.run()
  File "install/cluman_puppet.py", line 55, in run
    puppet_cmd.execute()
  File "util/command.py", line 106, in execute
    raise RuntimeError(' '.join(self.command) + ' returned ' + str(self.exitcode) + ' instead of ' + str(self.expected_exitcode))
RuntimeError: /usr/bin/puppet apply --detailed-exitcodes --debug --verbose --logdest /var/log/cluman/puppet-run.log --modulepath=/opt/cluman/puppet/modules --config /opt/cluman/puppet/puppet.conf /opt/cluman/puppet/nodes/node_repo.pp returned 6 instead of [0, 2]
2018-12-10 05:10:53,134 INFO [__main__.<module>] =====================
2018-12-10 05:10:53,134 INFO [__main__.<module>]  FAILURE
2018-12-10 05:10:53,134 INFO [__main__.<module>] ======== END ========
2018-12-10 05:10:53,134 INFO [__main__.<module>] To have the environment variable updated, please logout and login from all opened shell on the current system

Solution: Re-run the install.sh.

Case: Messages timed out when running Heap Dump of Policy Server (QNS)/Policy Director (LB) process on Policy Server (QNS)/Policy Director (LB) VM.

Condition: Taking heap dump of Policy Server (QNS)/Policy Director (LB) process on Policy Server (QNS)/Policy Director (LB) VM. Heap dumps taken results in full GC. This in turn results in application pause which causes message time out.

Solution: It is recommended to take the heap dump during Maintenance Window (MW).

Case: mongod process not running on both pcrfclients after fresh installation.

pcs status throwing not installed error as follows:

pcs status
Cluster name: cps
WARNING: corosync and pacemaker node names do not match (IPs used in setup?)
Stack: corosync
Current DC: pcrfclientXX (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum
Last updated: Thu Jan 17 14:09:38 2019
Last change: Thu Jan 17 12:27:52 2019 by root via cibadmin on pcrfclientXX

2 nodes configured
10 resources configured

Online: [ pcrfclientXX pcrfclientXX ]

Full list of resources:

 Resource Group: mongod
     arbitervip (ocf::heartbeat:IPaddr2):       Started pcrfclientXX
     sessionmgr-27721   (systemd:sessionmgr-27721):     Stopped
     sessionmgr-27717   (systemd:sessionmgr-27717):     Stopped
     sessionmgr-27727   (systemd:sessionmgr-27727):     Stopped
     sessionmgr-27017   (systemd:sessionmgr-27017):     Stopped
     sessionmgr-27718   (systemd:sessionmgr-27718):     Stopped
     sessionmgr-27719   (systemd:sessionmgr-27719):     Stopped
     sessionmgr-27720   (systemd:sessionmgr-27720):     Stopped
 Clone Set: PingIP-clone [PingIP]
     Started: [ AB-RT-pcrfclient01 AB-RT-pcrfclient02 ]

Failed Actions:
* sessionmgr-27721_start_0 on pcrfclientXX 'not installed' (5): call=12, status=Not installed, exitreason='',
    last-rc-change='Thu Jan 17 12:25:40 2019', queued=0ms, exec=41ms
* sessionmgr-27721_start_0 on pcrfclientXX 'not installed' (5): call=39, status=Not installed, exitreason='',
    last-rc-change='Thu Jan 17 12:36:59 2019', queued=0ms, exec=40ms
* sessionmgr-27717_start_0 on pcrfclientXX 'not installed' (5): call=40, status=Not installed, exitreason='',
    last-rc-change='Thu Jan 17 12:36:59 2019', queued=1ms, exec=77ms
* sessionmgr-27727_start_0 on pcrfclientXX 'not installed' (5): call=41, status=Not installed, exitreason='',
    last-rc-change='Thu Jan 17 12:36:59 2019', queued=0ms, exec=115ms
* sessionmgr-27017_start_0 on pcrfclientXX 'not installed' (5): call=42, status=Not installed, exitreason='',
    last-rc-change='Thu Jan 17 12:36:59 2019', queued=0ms, exec=151ms
* sessionmgr-27718_start_0 on pcrfclientXX 'not installed' (5): call=44, status=Not installed, exitreason='',
    last-rc-change='Thu Jan 17 12:36:59 2019', queued=0ms, exec=159ms

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: inactive/disabled

Solution: On pcrfclient, run crm_resource --clean command.

Case: When the pcrfclient and the sessionmgr VM's are rebooted, it is observed that some of the session managers go offline.

Reason: Some session managers are not able to connect to arbiter because of InvalidReplicaSetConfig: replica-set IDs do not match error. This is because the replica-set ID in arbiter and sessionmgr's does not match.

Solution: Perform the following steps to change database path in mongoConfig.cfg file:

1. Remove reporting replica-set.

   build_set.sh --report --remove-replica-set --setname replica-setID

   For example, build_set.sh --report --remove-replica-set --setname set03

2. Update mongoConfig.cfg file with the new database path.

3. Run the /var/qps/install/current/scripts/build/build_etc.sh script from the Cluster Manager to finalize mongoConfig.cfg file.

4. Create replica-set with force option.

   build_set.sh --report --create --setname replica-setID --force

   For example, build_set.sh --report --create --setname set03 --force

5. Verify the new database path in mongod instance.

Case: The warmup solution warms up the PS node(s) by internally initiating configured number of warm up messages prior to connecting to the load balancer node and processing external calls. The warm up messages are processed by policy engine like regular messages and sessions are created/deleted in session database. The actual response/request messages generated in response to incoming warm up requests are not sent towards Policy Director (lb) node, as there is no connectivity established towards Policy Director (lb). As Policy Director (lb) nodes accept traffic during processing node(s) warm up phase, it can result in some call failures (3004) towards DRA/Gateway.

Due to non-connectivity towards Policy Director (lb), if PS nodes are restarted (restartall.sh or individual restart) the following errors/warnings can be observed.

Caution

Executing restartall.sh will cause messages to be dropped.

Note	The errors/warnings are not seen after all the nodes are warmed up.

1. Some calls can be rejected with 3004 (DIAMETER_TOO_BUSY) error code.

2. The following Warnings/Errors can be seen in qns logs.

   WARN  c.b.d.p.event.DiameterMessageDealer - Unable to send message healthCheckHost.healthCheckRealm;1523905102;13492
                                        site-1-pps06  site-1-pps06 2019-02-13 09:41:06,775 [pool-2-thread-1] ERROR c.b.u.zmq.nodes.PushConnection - Exception sending message com.broadhop.exception.BroadhopException: No channels available
                                        at com.broadhop.utilities.zmq.nodes.PushConnection.getNextChannel(PushConnection.java:252) ~[com.broadhop.utility_14.0.1.r132522.jar:na]
                                        at com.broadhop.utilities.zmq.nodes.PushConnection.send(PushConnection.java:285) ~[com.broadhop.utility_14.0.1.r132522.jar:na]
                                        at com.broadhop.utilities.zmq.WorkerNode.send(WorkerNode.java:265) [com.broadhop.utility_14.0.1.r132522.jar:na]
                                        at com.broadhop.diameter2.policy.event.DiameterMessageDealer.processOutboundMessage(DiameterMessageDealer.java:671) [com.broadhop.diameter2.policy.endpoint_14.0.1.r132523.jar:n]
                                        at com.broadhop.diameter2.policy.event.DiameterMessageDealer.submit(DiameterMessageDealer.java:645) [com.broadhop.diameter2.policy.endpoint_14.0.1.r132523.jar:na]
                                        at com.broadhop.diameter2.policy.actions.SendDiameterResponse.execute(SendDiameterResponse.java:63) [com.broadhop.diameter2.policy.endpoint_14.0.1.r132523.jar:na]
                                        at com.broadhop.utilities.policy.async.PolicyLocalAsyncActionRunnable.run(PolicyLocalAsyncActionRunnable.java:33) [com.broadhop.utility_14.0.1.r132522.jar:na]
                                        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_72]
                                        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_72]
   
   ERROR c.b.policy.remote.jms.JmsReceiver - Error processing and deserializing incoming message com.esotericsoftware.kryo.SerializationException: Unable to deserialize object of type: com.broadhop.policy.remote.RemoteActionRequest
                                        at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:571) ~[na:na]
                                        at com.esotericsoftware.kryo.ObjectBuffer.readClassAndObject(ObjectBuffer.java:206) ~[na:na]
                                        at com.broadhop.policy.remote.jms.Jms.deserialize(Jms.java:271) ~[com.broadhop.policy.remote.jms_14.0.1.r131154.jar:na]
                                        at com.broadhop.policy.remote.jms.JmsReceiver.onMessage(JmsReceiver.java:170) ~[com.broadhop.policy.remote.jms_14.0.1.r131154.jar:na]
                                        at org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.java:1361) [activemq-all-5.9.0.jar:5.9.0]
                                        at org.apache.activemq.ActiveMQSessionExecutor.dispatch(ActiveMQSessionExecutor.java:131) [activemq-all-5.9.0.jar:5.9.0]
                                        at org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java:202) [activemq-all-5.9.0.jar:5.9.0]
                                        at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:129) [activemq-all-5.9.0.jar:5.9.0]
                                        at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:47) [activemq-all-5.9.0.jar:5.9.0]
                                        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72]
                                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72]
                                        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
                            Caused by: com.esotericsoftware.kryo.SerializationException: Unable to find class: com.broadhop.diameter2.actions.ICreateClientSession
   

Case: Session Limit Overload Protection is set to 400000 in Policy Builder. But when the calls are run at 2K TPS (CCR-I), the total number of sessions are growing till 475049 which is more than the allowed limit. However, CPS does not allow the sessions to grow after this, but still this is about 20% more than the allowed limit.

Mon Mar 4 09:55:52 UTC 2019

*** End-of-Collection ***
--------------------------------------------------------------------------------------------
Host Detail:
qns10,qns07,qns13,qns05,qns16,qns08,qns12
qns01,qns17,qns11,qns14,qns04,qns02,qns03
qns19,qns06,qns20,qns18,qns15,qns09
Measurement timer: 1    QNS Count: 20
--------------------------------------------------------------------------------------------
   Average    Success        TPS      Error  Time Used                       Messages
   16.9670       2100  2100.0000          0    35.6306              diameter_Gx_CCR-I
   13.1713       2005  2005.0000          0    26.4084                diameter_Rx_AAR
--------------------------------------------------------------------------------------------
   Average    Success        TPS      Error  Time Used                        Actions
    3.2845       6358  6358.0000          0    20.8826 com.broadhop.cache.impl.actions.GetSessionAction
    6.7885       2100  2100.0000          0    14.2558 com.broadhop.session.CreateEntry
    4.6131       2218  2218.0000          0    10.2319 com.broadhop.session.UpdateEntry
    0.6030       8551  8551.0000          0     5.1565                ResolveServices
    0.2736       8550  8550.0000          0     2.3397          ResolveServiceOptions
    0.1966       2223  2223.0000          0     0.4371           send.diameter_Gx_RAR
    0.1638       2223  2223.0000          0     0.3640           send.diameter_Rx_AAA
    0.1436       2100  2100.0000          0     0.3016         send.diameter_Gx_CCA-I
    0.0404       4231  4231.0000          0     0.1709 com.broadhop.policy.impl.actions.FormatTimeWithOffsetAction
    0.0148       8544  8544.0000          0     0.1266          BundleVirtualServices
    0.0181       4231  4231.0000          0     0.0766 com.broadhop.policyintel.impl.actions.StartPolicyReporting
    0.0049       4290  4290.0000          0     0.0211 com.broadhop.policyintel.impl.actions.StopPolicyReporting

Mon Mar 4 09:55:53 UTC 2019

*** End-of-Collection ***

^C
*** Exiting ***
[root@pcrfclient01 ~]# session_cache_ops.sh --count|grep Total;date
Total Number of Sessions  : 393478
Mon Mar  4 09:56:03 UTC 2019
[root@pcrfclient01 ~]# top_qps.sh
--------------------------------------------------------------------------------------------
Host Detail:
qns02,qns17,qns15,qns20,qns04,qns14,qns07
qns11,qns08,qns01,qns12,qns06,qns05,qns16
qns09,qns19,qns13,qns18,qns10,qns03
Measurement timer: 1    QNS Count: 20
--------------------------------------------------------------------------------------------
   Average    Success        TPS      Error  Time Used                       Messages
   15.4398       2100  2100.0000          0    32.4236              diameter_Gx_CCR-I
   16.1731       1848  1848.0000          0    29.8879                diameter_Rx_AAR
--------------------------------------------------------------------------------------------
   Average    Success        TPS      Error  Time Used                        Actions
    2.8085       6600  6600.0000          0    18.5362 com.broadhop.cache.impl.actions.GetSessionAction
    5.6463       2100  2100.0000          0    11.8572 com.broadhop.session.CreateEntry
    4.4516       2398  2398.0000          0    10.6750 com.broadhop.session.UpdateEntry
    0.5664       9000  9000.0000          0     5.0973                ResolveServices
    0.2597       9000  9000.0000          0     2.3369          ResolveServiceOptions
    0.1828       2398  2398.0000          0     0.4384           send.diameter_Gx_RAR
    0.1513       2398  2398.0000          0     0.3628           send.diameter_Rx_AAA
    0.1549       2100  2100.0000          0     0.3252         send.diameter_Gx_CCA-I
    0.0371       4500  4500.0000          0     0.1670 com.broadhop.policy.impl.actions.FormatTimeWithOffsetAction
    0.0137       9000  9000.0000          0     0.1234          BundleVirtualServices
    0.0166       4500  4500.0000          0     0.0747 com.broadhop.policyintel.impl.actions.StartPolicyReporting
    0.0047       4500  4500.0000          0     0.0214 com.broadhop.policyintel.impl.actions.StopPolicyReporting

Mon Mar 4 09:56:24 UTC 2019

*** End-of-Collection ***

^C
*** Exiting ***
[root@pcrfclient01 ~]# session_cache_ops.sh --count|grep Total;date
Total Number of Sessions  : 452341
Mon Mar  4 09:56:36 UTC 2019
[root@pcrfclient01 ~]# session_cache_ops.sh --count|grep Total;date
Total Number of Sessions  : 462917
Mon Mar  4 09:56:46 UTC 2019
[root@pcrfclient01 ~]# session_cache_ops.sh --count|grep Total;date
Total Number of Sessions  : 475049
Mon Mar  4 09:57:16 UTC 2019
[root@pcrfclient01 ~]# tailf /var/log/broadhop/consolidated-sessions.log
2019-03-04 15:13:45 - TPS_COUNT:                            SESSION_COUNT:                            LICENSE_COUNT: 200000
2019-03-04 15:15:15 - TPS_COUNT:                            SESSION_COUNT:                            LICENSE_COUNT: 200000
2019-03-04 15:16:45 - TPS_COUNT:                            SESSION_COUNT:                            LICENSE_COUNT: 200000
2019-03-04 15:18:15 - TPS_COUNT:                            SESSION_COUNT:                            LICENSE_COUNT: 200000
2019-03-04 15:19:45 - TPS_COUNT:                            SESSION_COUNT:                            LICENSE_COUNT: 200000
2019-03-04 15:21:15 - TPS_COUNT:                            SESSION_COUNT:                            LICENSE_COUNT: 200000
2019-03-04 15:22:45 - TPS_COUNT:                            SESSION_COUNT:                            LICENSE_COUNT: 200000
2019-03-04 15:24:15 - TPS_COUNT:                            SESSION_COUNT:                            LICENSE_COUNT: 200000
2019-03-04 15:25:45 - TPS_COUNT:                            SESSION_COUNT:                            LICENSE_COUNT: 200000
2019-03-04 15:27:15 - TPS_COUNT:                            SESSION_COUNT:                            LICENSE_COUNT: 200000
^C
[root@pcrfclient01 ~]# grep 'sessionLimitOverloadProtection' /var/broadhop/checkout/pcrfclient01-1/*
/var/broadhop/checkout/pcrfclient01-1/System-default-_GrTxALOLEeWR0MXs_g7BPA.xmi: sessionLimitOverloadProtection="400000">
[root@pcrfclient01 ~]# session_cache_ops.sh --count|grep Total;date
Total Number of Sessions  : 475049
Mon Mar  4 09:58:00 UTC 2019
[root@pcrfclient01 ~]# grep 'sessionLimitOverloadProtection' /var/broadhop/checkout/pcrfclient01-1/*;date
/var/broadhop/checkout/pcrfclient01-1/System-default-_GrTxALOLEeWR0MXs_g7BPA.xmi: sessionLimitOverloadProtection="400000">
Mon Mar  4 09:58:03 UTC 2019
[root@pcrfclient01 ~]# cat /etc/broadhop/qns.conf
QNS_COMMON_OPTS="
-Dcom.broadhop.run.systemId=system-1
-DapirouterContextPath=/ua/soap
-Dapi.ua.context.path=/ua/backend
-Dcom.broadhop.run.clusterId=cluster-scale
-Dcom.broadhop.run.instanceId=$HOSTNAME-$QNS_INSTANCE
-Dcom.broadhop.config.url=http://lbvip02/repos/run/
-Dcom.broadhop.repository.credentials.isEncrypted=true
-Dcom.broadhop.repository.credentials=qns-svn/3300901EA069E81CE29D4F77DE3C85FA@lbvip02
-Dcom.broadhop.referencedata.local.location=/var/broadhop/checkout
-DjmsRebalanceClients=true
-Denable.compression=true
-Denable.dictionary.compression=true
-DuseZlibCompression=true
-Dcom.broadhop.locking.autodiscovery=true
-DlookasideThreshold=3
-DcompressDebits
-Dnetworkguard.tcp.local=eth0
-DrefreshOnChange=true
-DenableRuntimePolling=true
-DdefaultNasIp=127.0.0.1
-Dua.version.2.0.compatible=true
-DsessionPadding=1200
-DnodeHeartBeatInterval=9000
-Dcom.mongodb.updaterIntervalMS=400
-Dcom.mongodb.updaterConnectTimeoutMS=600
-Dcom.mongodb.updaterSocketTimeoutMS=600
-DdbSocketTimeout=1000
-DdbSocketTimeout.balance=1000
-DdbConnectTimeout=1200
-DdbConnectTimeout.balance=1200
-Dmongo.client.thread.maxWaitTime=1200
-Dmongo.client.thread.maxWaitTime.balance=1200
-Dstatistics.step.interval=1
-Dmongo.connections.per.host=10
-Dmongo.connections.per.host.balance=12
-Dmongo.threads.allowed.to.wait.for.connection=12
-Dmongo.threads.allowed.to.wait.for.connection.balance=12
-DmaxLockAttempts=3
-DretryMs=3
-DmessageSlaMs=1500
-DshardPingLoopLength=3
-DshardPingCycle=200
-DshardPingerTimeoutMs=75
-Ddiameter.default.timeout.ms=1500
-DmemcacheClientTimeout=200
-Dlocking.disable=true
-Dcontrolcenter.disableAndsf=true
-DenableQueueSystem=false
-Dredis.keystore.connection.string=lb01:lb02:6379:6381
-Dcom.cisco.balance.dbs=6
-Dcom.cisco.balance.compression=true
-Duse.pre.v11.service.resolution=false
-Ddo.service.bundling.without.profiles=true
-Dpolicystate.optimize.inserts=true
-Dvirtualservice.optimize.crd=true
-Duse.ldap.vs.evaluation.order=true
-DPCRF_Name=TMOSITE1
-Djdiameter.accept.unknown_desthost=true
-Djdiameter.replace.unknown_desthost=true
-DmaxHash=2
-DdbSocketTimeout.cdrrep=1000
-DdbConnectTimeout.cdrrep=1200
-Dmongo.client.thread.maxWaitTime.cdrrep=1200
-Dmongo.connections.per.host.cdrrep=10
-Dmongo.threads.allowed.to.wait.for.connection.cdrrep=10
-DdbSocketTimeout.cdr=1000
-DdbConnectTimeout.cdr=1200
-Dmongo.client.thread.maxWaitTime.cdr=1200
-Dmongo.connections.per.host.cdr=10
-Dmongo.threads.allowed.to.wait.for.connection.cdr=10
-Dcisco.cdr.compression=true
-Dcisco.cdr.disableBlocking=true
-DapirouterContextPath=/ua/soap
-Dua.context.path=/ua/backend
-Dapi.ua.context.path=/ua/backend
-Dredis.keystore.connectionTimeout=20000
-DdbSocketTimeout.remoteBalance=1000
-DdbConnectTimeout.remoteBalance=1200
-Dmongo.client.thread.maxWaitTime.remoteBalance=1200
-Dmongo.connections.per.host.remoteBalance=12
-Dmongo.threads.allowed.to.wait.for.connection.remoteBalance=12
-Dmessage.buffer.early.processing.time=5
-Dlog.cdr.csv=true
-Dcom.broadhop.cdr.dir=/var/broadhop/cdr
-Dcom.broadhop.cdr.rollover.dir=/var/broadhop/cdr/rollover
-DuseV1ClientId=true
-DaddOnlysvcPlanAsVirtualService=true
-Dsk.db.replicateSessionSharding=true
-Dsk.db.audit.perShardTPS=1000
"
[root@pcrfclient01 ~]#

Solution: In Policy Builder, Session Limit Overload Protection must be configured such that there should be buffer/considerable difference between the configured value and the total system capacity. In session database, additional sessions are created beyond value configured in Session Limit Overload Protection until the Policy Server (QNS) process checks the current session count of the system in the next interval.

Issue: Diagnostics script output displays "application bundle or plugins failed to start".

Solution: During the start or initialization of application, it loads the configured data from the Policy Builder and brings up the bundles or plugins. The following table lists reasons and corresponding actions for solving the issue.

Perform the following steps to resolve the issue in HA environment:

1. Stop the QNS process running on the Policy Server (QNS) VMs.

   If this is a lab or no traffic is running, use stopall.sh script to stop all the QNS processes.

2. Drop the scheduler database from the ADMIN replica-set.

   use scheduler
   db.dropDatabase()
   

3. Clear the migration shards list from “cache_config” collection under sharding database from the ADMIN replica-set.

   use sharding
   db.cache_config.update({"_id" : 1},{$unset : {"migratingShards" : 1}})
   db.cache_config.update({"_id" : 2},{$unset : {"migratingShards" : 1}})
   

4. Set migration field to false and unset the prev_shard field in “buckets” collections under sharding database from the ADMIN replica-set.

   • For GR setup, collection name is the name of the bucket (for example, bucket_1 and bucket_2).

   • For SK DB in HA setup, collection name is the SK DB bucket name (for example, db.sk_buckets).

   • For SK DB in GR setup, collection name is the SK DB bucket name in GR setup (for example, db.sk_buckets_1 and db.sk_buckets_1).

   use sharding
   db.buckets.update({"prev_shard" : <x>}, { $unset: { "prev_shard" : <x>},$set :{"migration" : false} },{ multi: true })
   db.buckets.update({"prev_shard" : <x>}, { $unset: { "prev_shard" : <x>},$set :{"migration" : false} },{ multi: true })
   

   where, <x> is the output on the bucket to identify the prev_shard.

   • To find <x> in GR setup, execute db.buckets_1.find({“migration”:true}) & db.buckets_2.find({“migration”:true}) command.

   • To find <x> in HA setup, execute db.buckets.find({“migration”:true}) command.

   • In SK DB enabled setups,

     • To find <x> in HA, execute db.skbuckets.find({“migration”:true}) command.

     • To find <x> in GR, execute db.skbuckets_1.find({“migration”:true}) & db.buckets_2.find({“migration”:true}) command.

5. Start the QNS processes which are stopped in 1.

   If this is a lab or no traffic is running, use startall.sh script to start all the QNS processes.

6. Login to the OSGI console and use listshards command to list the shards.

7. If rebalance is still in running or InProgress state or rebalance is required, execute the following commands from OSGi console:

   • For GR setup,

     migrate <site_name>
     rebalance <site_name>
     

   • For HA SK DB,

     “migratesk“
     “rebalancesk“
     

   • For GR SK DB,

     migratesk  <site_name>
     rebalancesk <site_name>
     

8. Run diagnostics.sh command.

Issue: When Session Managers with memcached (using default UDP protocol) are rebooted while processing traffic, System Timeouts are observed.

Solution: To recover from this issue, you must can configure memcache to use TCP protocol by setting -Denable.memcache.on.tcp=true in qns.conf file.

Issue: MongoDB member is not coming up as Secondary after reboot. The member state as UNKNOWN when executing diagnostics.sh --get_replica_status command.

Condition: Sometimes after restart, the replica set members do not come up as expected. Few of them move to REMOVED state with "errmsg" : "Our replica set config is invalid or we are not a member of it", "code" : 93, and "codeName" : "InvalidReplicaSetConfig" when rs.status() is executed on the available member.

Possible Cause: This happens because during restart, when loading the configuration from the disk, MongoDB tries to discover itself from the configuration. It would iterate through each enlisted member from the configuration and execute a simple isSelf test command against every member. If the node cannot determine or resolve its own hostname in the course of this process then it would assume that it is not the part of the configuration file. At this point MongoDB does not retry to discover itself until it is restarted, or it receives a new config through rs.reconfig().

Case: User has extra lbvips which are not needed anymore and wants to remove the extra lbvips. Even after removing old lbvip using pcs resource delete RESOURCE_ID command, the old lbvips keep appearing in crm_mon -1 output. The following section provides the steps to remove the extra lbvips so that they don't come up again in the future.

Solution: For solution, see Remove Traces of Old Policy Director (LB) VIPs section in the CPS Operations Guide.

Issue: No Gx sessions are created and all incoming traffic is dropped.

The following message is printed in qns logs continously:

WARN c.b.d.p.event.DiameterMessageDealer - StaleSessionEndToEndSla breached at QNS, 
Time taken 1555DROPPING message RequestReceivedTime 1594136320622 currentTime 1594136322177

Analysis: When Stale Session Message Handling Configuration is configured in Policy Builder, request processing happens within the given SLA time period for the incoming request. The request or responses which cross the configured SLA are dropped.

Possible Cause: The incoming traffic is dropped if there is a clock skew on the Policy Server (qns) VMs from the Policy Director (lb) VM. If Stale Session Message Handling Configuration is configured, it is mandatory that all the VMs should be in time sync with the Policy Director (lb) VM.

Solution: Execute sync_times.sh ha command on the Cluster Manager to make sure all the VMs are in time sync with Policy Director (lb) VM.

Issue: If MongoDB processes on arbitervip are not coming up when enabling/disabling the MongoDB authentication.

Possible Cause: Due to corosync issue.

Solution:

1. Check where arbitervip is running and login to that host.

2. Execute ps -ef | grep mongod command.

3. Execute pkill mongod command.

4. Execute pcs resource cleanup command and wait for some time.

5. Check the status using pcs status command.

6. If the failed resource action entries still exist, again execute pcs resource cleanup command.

Issue: It takes longer time to publish the Policy Builder configuration in HA clusters.

Condition: SVN source and destination repositories are on different hosts/clusters rather than on the same host/cluster.

Solution: This is SNV server behavior and not CPS issue. If you are publishing on same host then use svn copy command and if host is different than use svn import command. As mentioned in the SVN docs, copy is faster than import.

For example, if you are logged in using http://lbvip02/repos/configuration and publishing to http://lbvip02/repos/run then both the hosts are same (lbvip02) and you can use svn copy command.

But if you are logged in using http://lbvip02/repos/configuration and publishing to http://<different_host>/repos/run then you can use svn import command.

SVN import takes more time than copy command. So, this is expected SVN server behavior.

The recommendation is that if you want to publish on different host or cluster, then open Policy Builder of other cluster and use other Cluster's run repository to publish.

1. Export policy configurations from hostA (clusterA) and push the same on hostB (clusterB) in /repos/configuration using SVN import command.

2. Open Policy Builder with other Cluster's IP address.

3. Login to Policy Builder with http://lbvip02/repos/configuration.

4. Publish to Cluster's to run repository using http://lbvip02/repos/run.

Issue: diagnostics.sh execution is stuck at one place for a longer time.

Analysis: This issue comes in the MongoDB replicas when the secondary replica lagging behind the primary replica. The reason is the higher CDR rate (approx. 8 k) in the reporting replica-set primary member. Due to this, diagnostics is stuck in hang state.

Scenario: To confirm whether the issue is due to reporting replica-set or not use the following steps.

1. Check the diagnostics by using diagnostics.sh --get_replica_status command.

2. If the diagnostics.sh --get_replica_status command execution is stuck, issue might be due to reporting replica-set. To cross verify, check whether the readonly user is able to login in to the MongoDB secondary member or not.

   mongo -u readonly -p ‘<password>' --authenticationDatabase admin sessionmgr<num>:<portNum>

3. Repeat 2 for all the secondary members.

4. If any of the readonly logins are stuck, check whether the inserts are high or not (approx. 8k) in the Primary member using mongostats command.

   mongostat -u admin -p '<password>' --authenticationDatabase admin --host sessionmgr<num>  --port <portnum>

Solution: To unblock the hanged secondary replica-set member, use the following steps:

1. Stop the sessionmgr mongod process by using /usr/bin/systemctl stop sessionmgr-XXXXX command.

   where, XXXXX is the database port number

2. Clear data directory of that sessionmgr by using \rm -fr <data directory path of that mongod> command.

3. Start the sessionmgr mongod process by using /usr/bin/systemctl start sessionmgr-XXXXX command.

   where, XXXXX is the same database port number mentioned in 1.

4. When this database member goes to 'SECONDARY' state, check the replica-set status by using diagnostics.sh --get_replica_status command.

   Note	If a member is shown in an unknown state, it is likely that the member is not accessible from one of other members, mostly an arbiter. In that case, you must go to that member and check its connectivity with other members.Also, you can login to mongo on that member and check its actual status.

After unblock the hanged secondary replica-set member. the load balancing and incoming CDR rate must be configured as a policy change in the Policy Builder based on the customer setup. For more information, contact your Cisco Account representative.

Issue: rebalance or migrate SK database commands are running but unable to complete.

Condition: Run rebalancesk from OSGi CLI. After hours, rebalancesk status shows following output instead of Rebalanced:

Remaining buckets: <number greater than 0>

Expected Output:

osgi> rebalanceskstatus
Rebalanced

Solution:

1. Login to the ADMIN replica-set PRIMARY member.

   mongo <sessionmgr_VM_hostname>:<Port_Number>

2. Check the records in the tasks collection.

   PRIMARY> use scheduler 
   PRIMARY> db.tasks.find()

3. If any tasks are present, remove those tasks.

   PRIMARY> db.tasks.remove({})

4. Connect to sharding database.

   use sharding

5. Run the following queries.

   db.sk_buckets.update({"migration": true},{'$set' : {"migration" : false }},{multi:true})
   db.sk_buckets.updateMany({},{ $unset: { "prev_shard": ""} })

6. Restart one of the qns service using monit restart qns-x command.

7. Once qns is up, execute rebalancesk command to rebalance the SK database shards.

   Example:

   OSGI> rebalancesk
   Rebalancing ...
   All versions up to date
   Obtaining lock
   Current version: xxx. Validating all instances on same version
   Min bucket size: xxxx
   Balance calculations completed successfully
   All versions up to date
   All versions up to date
   Obtaining lock
   Completed: 100%
   Migration completed successful

   Note	Once rebalance command is complete, execute migratesk command.

8. Execute migratesk command to migrate the SK database shards.

   Example:

   OSGI> migratesk
   Migrate ...
   All versions up to date
   Obtaining lock
   Completed: 100%
   Migration completed successfully

Issue: Import operation fails when user tries to import the same CRD through GUI and curl command immediately. The same MongoDB tables are changed which result in bad CRD state which is not recommended.

Solution: Since there is a limitation on updating json/bson files, -DuseMongoCLI=true configuration should not be used while exporting CRD.

Either set the parameter -DuseMongoCLI=false or remove the parameter from the setup where CRD export/import operations are being executed.

Issue: During import, Unauthorized User alert is displayed on the browser when performing import operation through GUI.

Solution: Clear the browser cookies when Unauthorized User alert is displayed while performing CRD import operation through GUI.

Issue: Frequent TCP connection reset seen in Policy Director (LB) nodes on port 3868, 3869, 3870 by HAProxy.

Analysis: User is running haproxy on Policy Director (LB) nodes with keepalive (check interval) for every 2 sec and forwarding data received on port 3868 to 3868 port on server lb01-A lb01:3868 and vice versa. But in tcpdump packet captured on LB01 VM, there is a connection reset on port 3868 every 2 seconds in a sequence: syn > syn,ack > rst,ack.

listen diameter-int1-vip
bind 172.17.50.128:3868
mode tcp
option tcpka
balance leastconn
server lb01-A lb01:3868 check
server lb01-B lb01:3869 check
server lb01-C lb01:3870 check
server lb02-A lb02:3868 check
server lb02-B lb02:3869 check
server lb02-C lb02:3870 check

The sequences of packets including reset is completely normal. This is the way haproxy performs health checks efficiently. As soon as haproxy has discovered that the endpoint is up, there is no point in wasting any further resources at either end. It turns out that using TCP RST is the most efficient way for kernels at both ends of the connection to finish their conversation and free up those resources. This is an expected behavior and no traffic loss is observed.

Issue: There is mismatch between given password and VM's password.

Condition: Encountered following error when running /var/qps/install/current/scripts/bin/support/manage_sshkey.sh script.

"Unable to check password"

Solution: Make sure root password is same for all CPS VMs.

Execute /var/qps/bin/support/change_passwd.sh from installer VM to change the password.

Issue: During creation of PCRF or UDC shards, if there’s an issue that isn’t monitored, the Admin sharding database gets created partially. This leads to an issue in the index creation and instance/configuration collection of sharding database.

Symptoms:

Policy Server (QNS) VM or UDC VM keeps restarting. This restarting of VMs leads to failure during shard creation.

OR

Rebuilding SK database operation is failing.

OR

There is only one index in shards collection of sharding database. You can confirm by logging into the Mongo Shell of Admin DB and checking the database shards.

mongo sessionmgr01:27721
set06:PRIMARY> use sharding
switched to db sharding
set06:PRIMARY> db.shards.getIndexes()
db.shards.getIndexes()
[
       {
              "v" : 2,
              "key" : {
                     "_id" : 1
              },
              "name" : "_id_",
              "ns" : "sharding.shards"
       }
]

Solution:

1. Login to Mongo Shell of Admin DB. Refer to /etc/broadhop/mongoConfig.cfg file to get the admin primary shard details.

   For example, mongo sessionomgr01:27721

2. Drop the shards collection using db.shards.drop() command.

   Note	If UDC shard has issue, UDC admin shard needs to be dropped and on udc01 VM. After shards are dropped, restart qns-1 process.

3. Restart qns01 process on any Policy Server (QNS) VM using monit restart qns-01 command.

Issue: Passwordless access not working between Cluster Manager and ESXi hosts after changing SSH keys using manage_sshkeys.sh script.

Solution: Execute /var/qps/install/current/scripts/deployer/support/jvalidate.py to synchronize the keys with ESXi hosts and restore passwordless access.

Issue: PCRFCLIENT VM is deployed and powered ON successfully but is not reachable from Internal CPS VMs. Communication works for Management/OAM interface properly.

Possible Cause: As internal IP is not reachable, Cluster Manager is not able to push puppet configuration of the respective node.

Condition: This issue is applicable to VMware based installation.

Security settings on Vswitch port connected to Internal Interface on PCRFCLIENT VM has:

• Allow MAC Address Change set to false

• Allow Forged Transmits set to false

Solution: For fixing the issue it is recommended to set security settings on Vswitch port connected to Internal Interface on PCRFCLIENT VM as:

• Allow MAC Address Change to true

• Allow Forged Transmits to true

Issue: An unreachable source is selected as a time source in chronyd.

Cause: When chronyd is configured with multiple time sources, it tries to select the most accurate and stable source for synchronization of the system clock. When the best source becomes unreachable, chronyd doesn't immediately switch to the next available best source in an attempt to minimize the clock error.

Chronyd lets the clock run free for as long as its estimated error (in terms of root distance) based on previous measurements is smaller than the estimated error of the next available source, and there is still an interval which contains some measurements from both sources. If the first source was better than the next available source, it can take hours before the available source is selected, depending on its polling interval.

Solution: This issue is resolved automatically depending on the configured NTP source.

Issue: As CRD is in BAD state, all CRD APIs (except import all, list and query) are blocked. User is not allowed to use the CRD APIs.

Analysis: When user tries to import CRD data using CPS Central > Import Custom Reference Data, CPS marks System-CRD is BAD. This is because underlying CRD schema is not compatible with CRD Data being imported. Hence, all CRD APIs (except import all, list and query), are blocked and user is not allowed to use them.

Possible Cause: In some conditions, though user made sure the underlying CRD schema is compatible with CRD data being imported but the system state is still doesn't recover from BAD.

Solution: Connect to admin database and update the isSystemBad flag to false manually. Refer to the following sample commands:

use admin
switched to db admin
> db.state.find()
{ "_id" : "state", "isSystemBad" : true, "lastUpdatedDate" : ISODate("2021-06-18T09:13:00.961Z") }
> db.state.updateOne({_id:"state"},{$set:{isSystemBad:false}})
{ "acknowledged" : true, "matchedCount" : 1, "modifiedCount" : 1 }
> db.state.find()
{ "_id" : "state", "isSystemBad" : false, "lastUpdatedDate" : ISODate("2021-06-18T09:13:00.961Z") }
>

Issue: Recurring quota creation or refresh does not happen when Recurrence Frequency is configured as Bill Cycle under Recurring Quota Template in the Policy Builder. This impacts new subscribers creation after migration to CPS 20.2 or later releases.

Possible Cause: When the Policy Builder with recurring quota template configured with Bill Cycle (RFAmt ignored) is imported into the CPS 20.2 or later release, the recurrence frequency defaults to Month(s) instead of Bill Cycle. This issue is applicable for Policy Builder based Balance templates and not for CRD based Balance templates.

for FILE in *; do `sed -i 's/ (RFAmt ignored)//g' $FILE`; done

Issue: After ISSM is complete, pcrfclient disk space and memory are getting filled.

Possible Cause: This issue occurs when the bulkstats files are copied during ISSM and are kept on hold by the collectd service even though they are removed from the system. This causes load on the system and ends up increasing the system storage and memory simultaneously.

Solution: Confirm if the collectd service is consuming high memory.

service collectd status | grep -i mem
Redirecting to /bin/systemctl status collectd.service
Memory:  70.2G

The memory is at 70+ GB which is not normal for collectd service running on pcrfclient VM.

Perform the following operations on pcrfclient 01 and 02 VMs. If your setup is a GR or a dual cluster, make sure that the following steps are performed on both sites pcrfclient VMs.

The following is an example of how to restart the collectd on pcrfclient01 VM. You must follow the same procedures on pcrfclient02 VM and on other site pcrfclient VMs if your setup is GR or a dual cluster.

1. SSH to pcrfclient01.

   ssh root@pcrfclient01

2. Stop collectd service.

   monit stop collectd

3. Confirm that the collectd service is no longer monitored and also the service is successfully stopped.

   monsum | grep -i collectd
   collectd                         Not monitored               Process

   systemctl status collectd

4. Once the service is successfully stopped, bring the collectd service back on.

   monit start collectd

5. Confirm that the service is up and running using monsum and systemctl commands.

   monsum | grep -i collectd 

   systemctl status collectd

Case: Traps were not generated after changing the graphite_default user password.

Issue: Traps are not generated for Gx and LDAP from gen-gx-drop-trap.sh and gen-ldap-trap.sh. The event log (/var/log/broadhop/scripts/gen-gx-drop-trap.log) shows There is no Gx Message traffic on $HOST VM error for Gx and There are no Ldap queries processing on $HOST VM error for LDAP even when the traffic is running on the setup.

Analysis: To have a common password for all the Grafana users, customer changed the default password for graphite_default user and updated /var/www/html/htpasswd. They missed to update /root/.graphite_default with the latest password which is being refered to access Graphite database.