Operations Overview
The Operation page enables you to access various interfaces and perform operations, maintenance, and troubleshooting activities. It assists system administrators and network engineers to operate and monitor the Policy Server.
The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
The Operation page enables you to access various interfaces and perform operations, maintenance, and troubleshooting activities. It assists system administrators and network engineers to operate and monitor the Policy Server.
DRA monitoring page under operations includes the following options:
DRA Peer Monitoring
DRA Binding Monitoring
DRA SLF Bindings
DRA Relay Connection
Grafana
DRA peer monitoring page displays the active peer endpoints (by default) for the cluster node. You can click the toggle for active/inactive peers to view the active or inactive peer endpoints.
The active and inactive peer monitoring screens have resize option for each column. You can use the scrollbar to view multiple values.
When the page is loaded, the Autorefresh checkbox is enabled by default which refreshes peers data every 30 seconds. You can stop this functionality by disabling the checkbox. After every refresh, the Data Last Refreshed field is updated with the locale time.
You can use the filter option to filter active and inactive peer endpoints. You can also view all event logs and peer details for specific active or inactive peer endpoints of the cluster node.
Pagination support is provided in active and inactive peer endpoints table data. A number of rows per page drop-down are displayed below each table which contains the different set of numbers indicating the number of rows which can be shown per page. This option enables you to perform the following tasks:
Select the number of rows to be displayed in each page.
Specify the page to which you want to navigate.
Through Pagination toolkit, you can view records count displayed in the webpage link out of total number of records fetched by API. This value changes as per the change in filtering of records through filter toolbar.
You can use the Close All option to close all the displayed popups. By default the Close All option is disabled. If you have many popups open, the Close All option gets enabled.
Step 1 |
In CPS DRA, navigate to DRA Peer Monitoring. |
||||||||||||||||||||||||||||||||||||||||||||
Step 2 |
Select the Filter by drop-down and click on any one of the following data options displayed:
In the Search field, enter a search value and click either the Search icon or press Enter to view details. Based on inputs the vDRA searches the entry and displays the output. You can also use Regex/Wildcard filter criteria to view peer endpoint details. For example, if the input value is "ndc2c*" then the search results displayed are ndc2c.gx.xom,ndc2c.gxx.com and so on. |
||||||||||||||||||||||||||||||||||||||||||||
Step 3 |
Enter a value in the Filter Peer Endpoints option. |
||||||||||||||||||||||||||||||||||||||||||||
Step 4 |
Click Toggle for Active Peers to view filtered active peer endpoints or Toggle for Inactive Peers to view filtered inactive peer endpoints. Under Active Peer Endpoints:
Under Inactive Peer Endpoints:
The following tables describe the details displayed under Peer Endpoints section:
You can use the refresh option provided next to the toggle for active/inactive peer endpoints to refresh the table data. You can enable the Auto-refresh checkbox to refresh data every 30 seconds. The Data Last Refreshed field displays time when data is fetched from server. |
Step 1 |
In CPS DRA, navigate to DRA Peer Monitoring. |
||||||||||||||||||||||||
Step 2 |
Click Toggle for Active Peers to view active peer endpoints or Toggle for Inactive Peers to view inactive peer endpoints. |
||||||||||||||||||||||||
Step 3 |
To view details of a particular peer, click Details. The following details are displayed:
If the Auto-refresh checkbox is enabled, the Data Last Refreshed field is displayed at the top of the Details dialog box of the selected peer. When you select Details modal, the Data Last refreshed field displays the time at which peers data was last refreshed. If the aAuto-refresh is performed when modal is opened, Data Last refreshed time in the modal is not updated and you have to re-open the modal to view the updated data. |
Step 1 |
In CPS DRA, navigate to DRA Peer Monitoring. |
Step 2 |
Click Toggle for Active Peers to view active peer endpoints or Toggle for Inactive Peers to view inactive peer endpoints. |
Step 3 |
To view event logs of a particular peer, click Event Logs. The Peer Status Logs is displayed. If the Auto-refresh checkbox is enabled, the Data Last Refreshed field is displayed at the top of the Event Logs dialog box of the selected peer. When you select Event Logs modal, the Data Last Refreshed field displays the time at which modal is opened. The data is not updated when the modal is opened. You have to re-open the modal to get the updated data. Event log data is independent of auto refresh data. |
CPS vDRA stores bindings in the mongo database. A binding database is needed to map search keys to PCRF binding information. Each binding has a search key and binding data associated with it.
You can access CPS vDRA binding information based on the following supported search keys:
IMSI
IMSI + APN
MSISDN
MSISDN + APN
IPv6
IPv4
Perform the following steps to view DRA binding details:
Step 1 |
In CPS DRA, navigate to DRA Binding Monitoring. |
||||||||||||||
Step 2 |
To view CPS vDRA binding information for a supported search key, click on any one of the following options displayed in the DRA Binding page:
|
||||||||||||||
Step 3 |
Enter the required value. The search button is enabled which when clicked displays the following binding details:
|
Step 1 |
In CPS DRA, navigate to DRA Binding Monitoring. |
||||||||||||||||||||||||||||
Step 2 |
Select a supported search key and provide an input value in the search input field. |
||||||||||||||||||||||||||||
Step 3 |
Click Search. CPS vDRA Bindings is displayed with two links for Gx Session ID and Details in each row. |
||||||||||||||||||||||||||||
Step 4 |
To view Gx session details, click Gx Session ID.
|
Step 1 |
In CPS DRA, navigate to DRA Binding Monitoring. |
||||||||||||||
Step 2 |
Select a supported search key and provide an input value in the search input field. |
||||||||||||||
Step 3 |
Click Search. CPS vDRA Bindings is displayed with two links for Gx Session ID and Details in each row. |
||||||||||||||
Step 4 |
To view details, click Details. The following details are displayed in a details popup:
|
This section describes how to view SLF Bindings details.
Perform the following steps to view SLF binding details:
In CPS DRA, navigate to DRA SLF Monitoring.
|
Step 1 |
Select Subscriber ID. |
||||||||||||||
Step 2 |
Enter a valid subscriber ID. |
||||||||||||||
Step 3 |
Click Search.
|
||||||||||||||
Step 4 |
Click Details.
|
Step 1 |
Select IMSI. |
||||||||||||||
Step 2 |
Enter a valid IMSI. |
||||||||||||||
Step 3 |
Click Search.
|
||||||||||||||
Step 4 |
Click Details.
|
Step 1 |
Select MSISDN. |
||||||||||||||
Step 2 |
Enter a valid MSISDN. |
||||||||||||||
Step 3 |
Click Search.
|
||||||||||||||
Step 4 |
Click Details.
|
You can monitor different relay connections to remote DRAs using the DRA Relay Connection option.
Perform the following steps to view relay connections:
Step 1 |
Navigate to DRA Relay Connection. |
||||||||||||||||||
Step 2 |
Select the Filter by drop down and click on any one of the following data options displayed:
|
||||||||||||||||||
Step 3 |
Enter a value in the Filter Relay Connections field. |
||||||||||||||||||
Step 4 |
Click Toggle for Active Relays to view filtered active relay endpoints or Toggle for Inactive Relays to view filtered inactive relay endpoints. The following table describes the details displayed under Relay Connections:
You can check the Auto-refresh checkbox to refresh data every 30 seconds. The Data Last Refreshed field displays time when data is fetched from server |
Perform the following steps to view relay details:
Step 1 |
Navigate to DRA Relay Connection. |
||||||||||||||||||||||||||
Step 2 |
Click Toggle for Active Relays to view filtered active relay endpoints or Toggle for Inactive Relays to view filtered inactive relay endpoints. |
||||||||||||||||||||||||||
Step 3 |
To view details of a particular relay connection, click Details. The following details are displayed:
If the Auto-refresh checkbox is checked, the Data Last Refreshed field is displayed at the top of the Details dialog box of the selected peer. When you select the Details modal, the Data Last Refreshed field displays the time at which data was last refreshed. If the Auto-refresh is performed when the modal is opened, Data Last refreshed time in the modal is not updated and you have to reopen the modal to view the updated data. |
Perform the following steps to view relay event logs:
Step 1 |
Navigate to DRA Relay Connection. |
Step 2 |
Click Toggle for Active Relays to view filtered active relay endpoints or Toggle for Inactive Relays to view filtered inactive relay endpoints. |
Step 3 |
To view event logs of a particular relay connection, click Event Logs. The Event Logs for Relay Key is displayed. If the Auto-refresh checkbox is enabled, the Data Last Refreshed field is displayed at the top of the Event Logs dialog box of the selected peer. When you select Event Logs modal, the Data Last Refreshed field displays the time at which modal is opened. The data is not updated when the modal is opened. You have to re-open the modal to get the updated data. Event log data is independent of auto refresh data. |
CPS vDRA stores audit logs based on modules. If you enable the debug or trace logging function, it stores detailed logs in the log file for all the subscribers for those modules. To overcome this situation, in the CPS 21.2.0 release, vDRA supports tracing function of incoming and outgoing messages for a single subscriber in PCAP format.
The main functions are:
vDRA captures diameter request and response messages across all vDRA sites for a single subscriber and stores all messages in configured DB in PCAP format.
Captures Request and Answer messages as received from peer and as sent to the peer on ingress and egress director, respectively.
Starts or Stops the message trace by getting subscriber identity, which is IMSI/MSISDN/IPv6.
Retrieves the PCAP based on subscriber identity. By default, vDRA stores PCAP in admin-db.
DRA configures any other mongo db uri.
Note |
Make sure that configured mongo-db uri is having reachability for directors and workers. |
Perform the following steps to view trace messages:
Step 1 |
In CPS DRA, click the tab. |
||||
Step 2 |
In the DRA Subscriber Trace/ Monitor area, click any one of the following options displayed to trace messages for a single subscriber:
|
||||
Step 3 |
Enter the required subscriber identity values for IMSI/MSISDN/IPv6.
|
||||
Step 4 |
Click the following radio button:
|
||||
Step 5 |
In the Stop Time field, enter the time to stop the trace for single-subscriber.
|
||||
Step 6 |
Click Start to start the message trace. |
||||
Step 7 |
Click Export Subscriber Data to download pcap files from DRA. |
In the audit log, vDRA captures traces across different containers such as Diameter-Endpoint, Binding and so on for success and failure messages. To overcome the difficulties of consolidating and getting the trace for complete session creation to termination from existing audit logs, vDRA captures subscriber-based logs based on IMSI/MSISDN/IPv6.
How it works:
Enable message trace in all possible sites where peer can connect. This is because, only those sites that are configured with trace detects the session.
Ingress director processing CCR-I/AAR maps the subscriber to the monitor session and notifies the start of session to all other sites. Thus subsequent messages for that session are captured in other sites. DSCP values are captured only for outbound messages.
On Start trace, other sites are notified with configured subscriber identity and any messages related to that configured subscriber identity/session are captured in other sites. On stop trace, other sites are notified with configured subscriber identity and capturing sessions are stopped for that configured subscriber identity. Other nodes in the system and other sites are notified on session monitoring through Control Plane advertisement.
vDRA captures Request and Answer message as received from peer and as sent to the peer on ingress and egress director, respectively.
After vDRA receives CCR-I for trace enabled subscriber, vDRA parses subscriber identity values (IMSI/MSISDN/IPv6) from CCR-I messages and stores them in internal cache along with session ids. This information is used for tracing initial AAR based on IPv6.
vDRA traces all other messages Gx CCR-U/T/RAR and Rx AAR update/RAR/STR based on session IDs stored on internal cache.
Following are the limitations:
DSCP value is captured only for outbound messages. For inbound messages, DSCP value is not captured.
For inbound messages, only diameter messages along with ip address and port are captured.
If same subscriber connects with different identity, then you need to configure a separate trace request for subscriber. DRA does not resolve the multiple identities for a subscriber automatically.
Enable message trace in all possible sites where peer can connect.
Supports only Gx, Rx application messages
By default, DRA can monitor maximum of 50 subscribers.
Message trace should be enabled only during maintenance window or in lab for any debugging purpose. By default, Trace is disabled.
DRA starts session trace only after receiving CCR-I message for that session. Until that even though message trace is configured, DRA does not capture trace for any messages related to that session.
The vDRA supports following functions to store live subscriber-based logs:
In vDRA, enable monitor activity in all possible sites where peer can connect. This is because, only those sites enabled with monitoring activity detect the session.
After monitor activity starts, vDRA captures and stores all possible logs related to the Policy Builder (PB), Customer Reference Data (CRD) configuration, Route selected, Shard selected, and traces of logs in different containers in monitor_activity_db DB.
If monitor activity stops, vDRA notifies other sites to stop capturing activity logs.
vDRA notifies other nodes in the system and other sites on session monitoring through Control Plane advertisement.
Once DRA receives CCR-I for monitor activity enabled subscriber, DRA parses subscriber values (IMSI/MSISDN/IPv6) from CCR-I messages and stores them in internal cache along with session ids. This information is used for monitoring initial AAR based on IPv6.
vDRA monitors all other messages Gx CCR-U/T/RAR and Rx AAR update/RAR/STR based on Session IDs stored on internal cache.
DRA monitors messages in Sd interface TSR, CCR-T.
DRA monitors messages in Gy interface CCR-I, CCR-T.
DRA monitors messages in Sy interface SLR, STR.
When monitor activity is enabled for a particular subscriber and set logger level of DRA.trace to trace, then DRA logs all activities of particular subscriber in a consolidated qns log under trace level.
vDRA monitors activity for both Gx/Rx and non Gx/Rx messages
vDRA monitors maximum of 50 subscribers.
By default, vDRA monitors subscriber activity for 24 hours.
By default, DRA stores monitor activity keys and activity logs in mongo-admin-a: 27017, mongo-admin-b: 27017, and mongo-admin-c: 27017.
By default, vDRA sets activity collection size as 1024 MB.
Monitoring non Gx/Rx protocol option is applicable for Monitor Activity and not for Messages Traces.
Following are the limitations:
Logs contain:
Monitor subscriber activity configurations-related to CRD and Policy Builder
Activity flows in each container
Shard selection logs
Routing information
DRA starts session monitoring only after receiving CCR-I message for that session. Until that even though monitor activity is configured, DRA does not capture activity for any messages related to that session.
Does not support monitoring activity of the same subscriber with multiple times.
Recommended to use only during debugging session or testing.
Recommended only 10 subscribers during live traffic. By default, it is in disabled state.
In CPS DRA, click the
tab.In the DRA Subscriber Trace/ Monitor area, click any one of the following options displayed to tmonitor subscriber activities for a single subscriber:
IMSI
MSISDN
IPv6
Enter the required subscriber identity values for IMSI/MSISDN/IPv6.
Click the following radio button:
Parameter |
Description |
---|---|
Monitor Activity |
Click the Monitor Activity radio button to monitor subscriber activity in the vDRA. For more information about configuration parameters, refer the dra subscriber-monitor-activity db-connection and dra subscriber-monitor-activity db-activity-collection-max-size CLI Commands in the CPS vDRA Operations Guide. |
In the Stop Time field, enter the time to stop the monitor activity for single-subscriber.
Supports date/time formats:
yyy-MM-dd HH:mm::ss – Supports date and time format in ISO format.
HH:mm:ss – Allows to enter only time where DRA internally considers date as current date.
By default, DRA monitors each subscriber for 24 hours duration and after that DRA automatically stops the monitoring subscriber activity.
Manually you can change the duration before starting the monitor activity.
Click Start to start the monitor activity.
Click Export Subscriber Data to download subscriber activity files from DRA.
Use the monitor subscriber-activity CLI command to monitor live subscriber activity logs in the vDRA. This monitor subscriber-activity CLI command is used only to view live logs. For example, use the following Syntax to view subscriber identity logs:
monitor subscriber-activity imsi <IMSI value> user <admin>
monitor subscriber-activity msisdn <MSISDN value> user <admin>
monitor subscriber-activity ipv6 <IPv6 value> user <admin
For more information, see monitor subscriber-activity section in the CPS vDRA Operations Guide.
Note |
This feature has not been validated for all customer deployment scenarios. Please contact your Sales Account team for support. |
CRD, Metadata DB connectivity, and Consul failures lead to improper processing of diameter messages in the Worker node. To enhance product resiliency in the Worker CRD failure, Worker Metadata DB, and Worker consul readiness scenarios, vDRA supports health checks to ensure that all prerequisites are met before traffic is accepted by nodes.
As a best practice, the following validation is done during Worker or Diameter node startup:
CRD Validation for Diameter and Binding Initiation: Applicable only for binding node:
During container startup (Diameter or Worker node), vDRA performs the CRD validation, and if the CRD is not loaded properly, then the container is marked as unhealthy.
During runtime state, if there’s any validation that is performed for CRD update and if there are any errors during CRD cache loading, that node is revoked from IPC message processing and marked as unhealthy.
Metadata DB Access Check from Binding Node: Applicable for Worker node. During container startup the metadata DB connection validation is performed. If there’s any failure/issue, then the node is marked as unhealthy and gets removed from the IPC message processing.
During runtime, if there are any errors during Metadata DB cache reloading, that node gets revoked from IPC message processing and marked as unhealthy.DRA App Configuration Validation: Applicable only during application startup in Worker/Director node.
Use the consul-health
CLI configuration command to monitor consul failures during startup. This script is part of supervisorctld
.
Equal Priority on Health Checks: Use the system-config get-cpu-priority and system-config set-cpu-priority CLI commands to ensure that health checks for critical processes have equal priority to the monitored entity process
You can access the Grafana interface under DRA Monitoring to monitor installation. It is a third-party metrics dashboard and graph editor. Grafana provides a graphical or text-based representation of statistics and counters collected in the Prometheus database.
Note |
After the DRA Director (DD) failover/reboot, the TPS values in Grafana dashboards takes approx. 5 minutes to fetch and display the latest updated values. Until the values are updated, Grafana displays the old data. |
For more information about Grafana in vDRA, refer to the Prometheus and Grafana chapter in the CPS vDRA Operations Guide.
API information option enables you to view API related information:
Service Orchestration API: to manage Policy Builder data
Select the link to view the documentation and usage examples.