This section describes how to debug the issues that may occur when you deploy PCF through the SMI Deployer.

To debug the deployment issues, use the following checklist. If the checklist does not assist you in resolving the issue, analyze the diagnostic data that is available in the form of logs.

kubectl get services -n namespace 

firewall-cmd –zone=public –add-port= port/tcp –permanent 

nc -v 

This section describes how to refresh the PCF Ops Center to display the latest configurations.

Problem

The PCF Ops Center is not considering the recent configurations due to which you may observe stale data or not get the expected response.

Resolution

You can refresh the PCF Ops Center using the basic and advanced steps. Perform the advanced steps only when the basic steps do not resolve the issue.

Basic

1. Run the following to undeploy PCF from the Ops Center:

   system mode shutdown 

2. Use the following to manually purge any pending deployments from the helm:

   helm delete --purge helm_chart_name 

3. From the master node, run the following to delete the configMaps from the namespace where PCF is installed:

   kubectl delete cm config_map_name -n namespace 

4. Run the following to delete the product-specific configMaps from the CNEE namespace.

   1. Use the following to list the available configMaps:

      kubectl get configmaps -n namespace 

      From the list, determine the configMap that you want to delete.

   2. Run the following to delete the configMap:

      Kubectl delete configmap configmap_name -n namespace 

5. Use the following commands to reinstall the helm chart. Once the chart is installed, a new instance of the PCF Ops Center is available.

   helm upgrade –install release name addR/chart_name -f filenames --namespace namespace 

Advanced

1. Remove the cnee-ops-center.

2. Delete the configMaps from the namespace.

   For more information on step 1 and 2, see the Basic steps.

3. Install the PCF Ops Center. For information on how to PCF Ops Center, see Deploying and Accessing PCF.

kubectl describe pod pod_name -n namespace 

kubectl logs -n namespace consolidated-logging-0 

This section describes how to resolve the issues that report the Subscriber Not Found or Primary Key Not Found messages.

Problem

When the NFs cannot find the subscriber details, they send the Subscriber Not Found or Primary Key Not Found to PCF.

Resolution

1. Analyse the logs of the PCF Engine and REST endpoint pod for the subscriber or primary key related issues.

   On the master node, run the following command to determine the engine and rest-ep pod.

   kubectl logs -n namespace pod_name 

2. Navigate to the pods and review the subscriber availability status and the subscriber count in the database. Based on the subscriber's status, take the appropriate action to resolve the issue.

   cdl show session count/summary 

This section describes how to troubleshoot the message routing issues.

Problem

You may observe a message routing failure when a message from the PCF endpoint incorrectly routes a message from Canary to the PCF Engine. The issue occurs when the message is sent to an incorrect PCF group.

Resolution

The following conditions might be causing the message routing failure. Check for these conditions and correct them, if necessary.

• From the PCF Ops Center, manually verify that the routing rules are configured correctly and they match the incoming traffic.

• Ensure that the Istio proxy is injected in the pcf-rest-ep pod.

• Verify that the virtual services are generated using the istioctl command. For more information on the traffic routing logs, see Collecting the Troubleshooting Information.

• Enable the DEBUG level for com.cisco.pcf.endpoint.routing and review the pcf-rest-ep logs for any issues. Use the following command to enable the DEBUG level:

  debug logging logger com.cisco.pcf.endpoint.routing level debug 

Alerts

If you encounter issues in your PCF environment, gather and analyse the information associated to the failed action or process. Having this information enables you to detect the component that experiences the failure and resolve the issue faster.

The following table covers the components which might experience an issue, and the logs that contain the information corresponding to the issue.

helm get namespace -pcf-rest-ep 

kubectl logs -n namespace pod_name 

show full-configuration 

show running-config 

kubectl get pod -o yaml -n namespace pcf-rest-ep pod_name 

istioctl get virtualservice -n namespace -o yaml 

istioctl get destinationrules -n namespace -o yaml 

cdl show session count/summary 

Alerts are notification messages that are generated when incidents requiring your attention or response occur. Review the historical and active alerts to determine the issue.

Alerts for PCF are generated through the CEE utility. To view these alerts, run the following command in the CEE Ops Center:

For active alerts:

show alerts active 

For historical alerts:

show alerts history 

Note	You must have appropriate permission to view the alert details.

For information on application-based alerts, see PCF Application-Based Alerts.

This section describes the codes that PCF reports for the interface errors.

Interface codes are generated as part of the logs or captured in the statistics.

The following tables describes the error and the corresponding codes:

Note	The generic error codes are applicable for all the network interfaces.

This section describes how to enable PCF to forward the logs to the Splunk server.

Splunk is a third-party monitoring application that stores the log files and provides index-based search capability. You can configure PCF to send the logs securely to a Splunk server which could be an external server.

Important

The Splunk server is a third-party component. Cisco does not take the responsibility of installing, configuring, or maintaining this server.

Use the following configuration to forward the logs to the Splunk server.

config 
  debug splunk 
    batch-count no_events_batch 
    batch-interval-msbatch_interval_ms 
    batch-size-bytes batch_size 
    hec-tokenhec_token 
    hec-url hec_url  
    end 

The following is an example configuration:

configure
 debug splunk hec-url https://splunk.10.86.73.80.nip.io:8088
 debug splunk hec-token 68a81ab4-eae9-4361-92ea-b948f31d26ef 
 debug splunk batch-interval-ms 100
 debug splunk batch-count 10
 debug splunk batch-size-bytes 102400
 end 

NOTES:

• debug splunk —Enters the configuration debug mode.

• batch-count no_events_batch —Specify the maximum number of events to be sent in each batch.

• batch-interval-ms batch_interval_ms —Specify the interval in milliseconds at which a batch event is sent.

• batch-size-bytes batch_size —Specify the maximum size in bytes of each batch of events.

• hec-token hec_token —Specify the HTTP Event Collector (HEC) token for the Splunk server.

• hec-url hec_url —Specify the protocol, hostname, and HTTP Event Collector port of the Splunk server. The default port is 8088.

This section describes how to ensure that the pods are running when PCF is upgraded.

Problem

When the PCF version is upgraded to the subsequent available version, some pods such as CRD and Policy Engine stop running.

Resolution

Whenever you configure PCF ensure that you configure the following parameters:

• db global-settings db-replica replica_count

• db spr shard-count shard_count

• rest-endpoint ips ip_address1, ip_address2, ip_address3

• rest-endpoint port port_number

• engine engine_name

  replicas replica_count

  unified-api-replicas api_replica_count

  subversion-run-url repository_url

  subversion-config-url configuration_url

  tracing-service-name service_name

• service-registration profile locality profile_name

• service-registration profile plmn-list [ mcc mnc ]

• service-registration profile snssais [ sst sd ]