Table Of Contents
Cisco Service Control Business Intelligence Solution Guide,
Release 3.8.xMobile Vendor Specific Attributes (VSA)
Cisco SCA BB Reporter Features
Reports based on Topper/Aggregator (TA) Adaptor
Reports based on Real-Time Aggregating (RAG) Apapter
Cisco Insight v2 Reporting Tools
Enabling Business Intelligence for Video and HTTPS Reports by Using SCA Reporter
Obtaining Documentation and Submitting a Service Request
Cisco Service Control Solution Guide
Cisco Service Control Business Intelligence Solution Guide,
Release 3.8.x
Revised: Septemer 17, 2012, OL-26813-01![]()
Note
This document supports all 3.8.x releases.
1 Overview
The Cisco Service Control Business Intelligence (BI) solution provides the enhanced analysis and reporting of subscriber network traffic. The infrastructure of the Cisco Service Control BI solution consists of features that enables you to analyze the behavior of subscribers within the network. For example, the traffic analysis can include volume consumption, access patterns, top content providers, and usage trends. The Cisco Service Control BI solution offers tools for trend analysis, traffic comparison, and enhanced data reporting. It also provides ways to access this data over longer periods of time (up to one year).
Within the Cisco Service Control BI solution infrastructure, the Cisco Service Control Engine (SCE) sends Raw Data Records (RDR) to the Collection Manager (CM). The Collection Manager then uses software modules to perform various data aggregations and writes the data to an external database (for example, MySQL). The Cisco Service Control Application for Broadband (SCA-BB) Reporter and the Cisco Insight Reporter processes the data from the database and presents the data in web reports. Figure 1 illustrates the network topology of the Cisco Service Control BI solution.
Figure 1 Cisco Service Control BI Solution Network Topology
![]()
Traffic Database—MySQL, Sybase, or Oracle.
Operating System—Red Hat Linux, CentOS Linux, or Solaris.
For details on Operating Systems and Databases supported by Cisco Insight, see http://www.cisco.com/en/US/partner/docs/cable/serv_exch/serv_control/broadband_app/insight/rel30/user_guide/insightugv3.html.
Related Publications
Use this guide with the following Cisco documentation:
•
Cisco Service Control Management Suite Collection Manager User Guide
•
Cisco Service Control Application for Broadband User Guide
•
Cisco Service Control Application for Broadband Reference Guide
•
Cisco Service Control Application Reporter User Guide
•
Cisco Service Control Mobile Solution Guide
2 Features
The Cisco Service Control BI solution infrastructure generates a large amount of data that needs to be collected, processed, and stored. Reports on this data need to be generated within a reasonable response time to meet user requirements. The need for reporting on both detailed and long-term trends can create challenges. To address these challenges, the Cisco Service Control BI solution provides these features:
•
Cisco SCA BB Reporter Features
Collection Manager Features
The Collection Manager uses several complementary features to provide a compromise between level of detail and long-term storage of the data:
Focus on Significant Data
The term ClickStream describes the actual Hypertext Transfer Protocol (HTTP) requests that a particular subscriber triggers. The SCE filters out irrelevant URLs that secondary HTTP requests trigger. The ability to classify HTTP requests as belonging to the ClickStream of the subscriber allows an accurate and effective extraction of the web browsing habits of the subscriber. The ClickStream of the subscriber contains significant data about the access pattern of the subscriber. The ClickStream events make up only 1 to 5 percentage of the total HTTP requests, hence, the quantity of data that requires analysis also reduces.
You can set the volume threshold within the reports based on the ClickStream, and transactions for particular applications that exceed the threshold are filtered out. This filtering significantly reduces the amount of data that requires processing, without compromising accuracy.
Database Table Partitioning
The Collection Manager partitions records of the same type into separate tables within the database. These partitions are calculated according to time-stamp ranges. A rolling window mechanism is then used to delete the oldest partition periodically.
String Removal
Strings require storage space and have limited use in data mining. Eliminating strings frees up space for more storage. The Java Database Connectivity (JDBC) adapter of the Collection Manager can replace strings with empty strings based on the dbtables.xml configuration file. Eliminating strings results in a smaller overall database.
To modify a string field in the configuration file, use the <options> subtag in the <field> tag to overwrite the string with an empty value.
If INFO_String is removed, the following reports become unusable:
•
Top Email Account Owners.xml
•
Top Email recipients.xml
•
Top Email senders.xml
•
Top Newsgroups.xml
•
Top Subscriber To Newsgroup.xml
•
Top Peer-to-Peer (P2P) File Extensions.xml
•
Top Session Initiation Protocol (SIP) Domains.xml
If ACCESS_String is removed, the following reports become unusable:
•
Real-Time Streaming Protocol (RTSP) Host Distribution by Subscriber Packages.xml
•
Top HTTP Streaming Hosts.xml
•
Top RTSP Hosts.xml
•
Top Web Hosts.xml
•
Web Host Distribution by Subscriber Packages.xml
Aggregation of Usage Data
The aggregation process is run for each table as a database stored procedure. Aggregation of usage data significantly reduces memory requirements by freeing space previously used to store more granular data points. Aggregation is disabled by default.
The xUR (where x is a particular type of Usage RDR) table records are displayed over time. With little impact to the overall accuracy of the report, usage data can be aggregated for the following timeframes:
•
No aggregation for the first day
•
15-minute aggregations up to the three-month mark
•
1-hour aggregations up to the one-year mark
•
One-day aggregations if over one year
Adaptive Frequency Counts
Long-term trend reports are based on frequency counts, that is, how frequently a host was accessed. Performing frequency counts on an infinite data stream could be a complex computational task. To reduce the complexity of this task, the CM uses an adaptive frequency count method. In this method, the data for each of the most frequent events is aggregated and then stored. You can configure aggregation periods based on hour, day, or week.
Cisco SCE Features
Cisco SCE data classification features include:
•
Signature support—Support for over 700 application signatures.
•
Zero-day detection—Heuristic approaches for classification of application categories (VoIP, P2P, gaming).
•
Protocol packs—Updates the latest protocol signatures to the Signature Engine every two months.
•
External signature editor—Signature utility for creating L7 classification.
When a flow does not match a protocol signature, advanced classification mechanisms are used:
•
Behavioral classification—Flows of certain application categories usually have a distinct behavioral pattern (sparse or dense, unidirectional or interactive).
•
Classification based on recent history—Adjacent flows with similar source or destination are classified together because these flows usually belong to the same application.
•
Multistage classification—Accurate classification requires several packets. Therefore, temporary classification is used when immediate policy decision is needed.
Raw Data Records (RDR) on SCE
SCE platforms running SCA BB generate and transmit Raw Data Records (RDRs) that contain information relevant to the service provider. RDRs contain a wide variety of information and statistics, depending on the configuration of the system. RDRs are transmitted using a Cisco proprietary protocol. This requires you to use the Cisco Service Control Management Suite (SCMS) Collection Manager or to develop software to process the RDRs.
The data in some RDRs can also be exported using the NetFlow reporting protocol. NetFlow reporting allows the SCA BB solution to be more easily integrated with your existing data collectors.
The following are the main categories of RDRs:
•
Usage RDRs—Generated periodically. These RDRs contain the state of the usage counters, per service and per accounting scope. There are four types of usage RDRs:
–
Link Usage RDRs—Global usage per service, for the entire link.
–
Package Usage RDRs—Usage per group of subscribers, per service.
–
Subscriber Usage RDRs—Usage per subscriber, per service. These RDRs are generated for all subscribers. The Cisco Service Control Management Suite (SCMS) Collection Manager (CM) and Cisco Service Control Application (SCA) Reporter use these RDRs to generate top-subscriber reports and aggregated usage billing records.
–
Real-Time Subscriber Usage RDRs—Generated for selected subscribers only. The SCMS-CM and SCA Reporter use these RDRs by to generate detailed subscriber activity reports.
–
Zone Usage RDRs—Generated for configured zones, for subscriber usage in that particular zone.
•
Media RDRs—
•
Transaction RDRs—Generated for a sample of the flows. These RDRs are used to create statistical histograms such as Top TCP Ports.
•
Transaction Usage RDRs—Generated for every flow according to user-defined filters. These RDRs contain detailed Layer 7 information for browsing, streaming, and voice flows. They are used for flow-based billing.
•
Real-Time Signaling RDRs—Generated to indicate specific network events such as flow start or end. These RDRs are used to signal external systems to allow real-time actions across the network.
•
Malicious Traffic RDRs—Generated to indicate that the SCE platform has detected a traffic anomaly, such as a DDoS attack. These RDRs are used to detect attacks and attackers to mitigate them.
For details on various RDRs, see Cisco Service Control Application for Broadband Reference Guide and Cisco Service Control Application for Broadband User Guide.
Mobile Vendor Specific Attributes (VSA)
Vendor Specific Attributes (VSA) are RADIUS or Diameter attributes that are sent mostly in mobile environments. They can be captured from the traffic processed by the SCE and then reported to the billing server. They can also be sent to the Collection Manager in various RDRs to be used to generate a number of common mobile reports.
Capturing VSAs
VSAs can be captured by either of the following entities:
· SCE platform
· RADIUS Login Event Generator (LEG)
Only the VSAs that you select are captured. You can choose which attributes are relevant, and store and process only those attributes. You can select a maximum of 20 attributes from the list of available VSAs. The list of attributes to capture can be modified while the system is running.
For details on Mobile VSA, see the Cisco Service Control Mobile Solution Guide. Appendix B of the guide provides a list of supported Mobile VSAs. Chapter 6, Capturing and Reporting Subscriber Attributes of the guide provides details on capturing and reporting VSAs.
Cisco SCA BB Reporter Features
This section provides details on the Cisco SCA BB Reporter features for BI solution.
Reports
The Cisco Service Control BI solution includes enhanced reports that provide a comprehensive view of the network activity.
The Topper/Aggregator (TA) and the Real-Time Aggregating (RAG) adapters in the Collection Manager collects and processes these reports:
•
Bandwidth
•
Capacity
•
Subscriber demographics
•
Server activity
•
Traffic comparisons between regions or geographies, types of traffic, and so on
Reports based on Topper/Aggregator (TA) Adaptor
•
Top Subscribers
•
Top P2P Consumers/Uploaders/Downloaders
•
Top VoIP Subscribers
•
Top Video Consumers
•
Top Web Consumers
•
Cumulative Distribution of Subscriber Usage
•
Average Subscriber Consumption
Reports based on Real-Time Aggregating (RAG) Apapter
Reports from Video and Web groups falling under the following level 2 topics:
•
Providers
•
Hosts
•
Consumers
Cisco Insight Features
Cisco Insight v2 is a software platform based on a web 2.0 user experience standard, designed to collect and present reports, charts, and statistics about the traffic data collected by Cisco SCE devices.
Cisco Insight v2 is an independent component of the SCA solution, but still has some dependencies on the database schema of the deployed SCA Collection Managers and SCE devices. The Cisco Insight v2 application should be installed on a dedicated high-end device. However, for small deployments or demos, it could share the same device hosting the Cisco Collection Manager and its database, where traffic data is stored.
Cisco Insight v2 supports all reports available in SCA BB Release 3.5.0, 3.5.5, 3.6.0, 3.6.5, and 3.7.0.
Cisco Insight v2 Reporting Tools
Cisco Insight v2 provides a series of reporting tools that enables you to run interactive reports, save and share report definitions, schedule recurring reports, view and customize a dashboard, and share results. The Report Topics tab in Cisco Insight GUI provides various report topics available on the official Cisco Insight v2 template.
The report topics are organized in various report groups. Table 1 provides details of the report groups.
For a list of all report topics and details on each topic, see the latest Cisco Insight Reporter User Guide.
Reports
Cisco Insight reports are based on concepts such as leading input switch and series comparison, time controller, related reports, drill-down, and personal dashboard. Cisco Insight provides enhanced reports including these:
•
Network activity reports
•
Subscriber flows drill-down reports
•
Drill-Down by service reports
•
Video and HTTP Trends reports
Subscriber Flows Drill-Down Reports
You can drill down through the reports that display the open subscriber flows, that is, display top services, then display subscribers per service, and then display detailed information about a specific subscriber. Figure 2 illustrates the data flow between components during a drill-down to flow-per-subscriber data.
![]()
Note
The FUR RDRs must be enabled on SCE to activate the Subscriber Flow drilldown report on Cisco Insight.
The drill-down follows this data flow:
Step 1
Upon drill-down to a specific subscriber, Cisco Insight triggers a login operation to the Subscriber Manager (SM) via proprietary remote procedure call (PRPC) over the Subscriber Manager application programming interface (API). The login operation sets the monitor property value.
Step 2
The Subscriber Manager forwards this update to the Cisco SCE.
Step 3
The Cisco SCE generates flow-related RDRs for the subscriber and sends them to the Collection Manager.
Step 4
The Collection Manager JDBC adapter inserts the flow-related RDRs into the FUR table of the external database.
Step 5
Cisco Insight polls the FUR table in the external database and displays the flow information (Figure 2).
Figure 2 Data Flow of Drill-Down to Flow-Per-Subscriber Using Cisco Insight
![]()
Video and HTTP Trend Reports
The Cisco Service Control BI solution includes enhanced reports on Video and HTTP domains:
•
Service-related reports—Video service distribution, top Flash video hosts
•
Subscriber-related reports—Top video consumers, top browsing consumers
•
Provider-related reports—Top video providers, top web hosts
•
Trend reports—Changes over time
The SCE sends Video and HTTP TURs to the CM. The CM then routes these TURs to its RAG adapter. The RAG adapter:
•
Aggregates these TURs to the domains at the first aggregation level.
•
Periodically populates the top domains into a new table in the database schema.
A scheduled database server aggregation job periodically aggregates the first-level aggregation data into the second aggregation level. A reporter queries the database to generate the Video and HTTP trend reports.
3 Enabling Business Intelligence for Video and HTTPS Reports by Using SCA Reporter
This section provides details on enabling Business Intelligence for Video and HTTPS Reports. Enabling Business Intelligence involves:
•
Configuring the Collection Manager
•
Configuring the Cisco SCA Reporter
Configuring the Collection Manager
Step 1
Confirm that the RAG adapter is enabled.
If required, uncomment the following line in ~/cm/config/cm.conf file:
adapter.4=com.cisco.scmscm.adapters.rag.RAGAdapter
Step 2
Confirm that the RAG adapter is configured with sufficient heap size. Increase the value (for example 2048 MB here) in ~/cm/config/cm.conf according to sizing considerations:
com.cisco.scmscm.adapters.rag.RAGAdapter=-Xmx2048M
Step 3
Confirm that the RAG adapter is configured to perform the aggregations.
Copy ~/cm/config/ragadapter/repository/http_TURs.xml and/or ~/cm/config/ragadapter/repository/video_TURs.xml to ~/cm/config/ragadapter/
Step 4
(Optional) Confirm that the data reduction is enabled on RPT_TOP_HTTP_DOMAINS, RPT_TOP_HTTP_HOSTS, RPT_TOP_VIDEO_DOMAINS, and RPT_TOP_VIDEO_HOSTS.
a.
If required, update ~/db_maint/dbperiodic.conf.
b.
Rerun ~/db_maint/create_periodic_del_procs.sh.
c.
Rerun ~/scripts/dbperiodic.py.
Configuring the Cisco SCA Reporter
Step 1
Confirm that the Video and Web monitoring groups of report enabled. If required, uncomment these lines in C:\Program Files\Cisco SCA\SCA Reporter 3.7.0\plugins\com.cisco.p3.apps.scasbbtemplates_3.7.0\templates\repository.xml file:
<group folder="Video Monitoring"/>
<group folder="Web Monitoring"/>
Configuring the Cisco SCA BB
Step 1
Assign counters to the ClickStream-New Page and ClickStream-New Site services.
Step 2
Enable Video TURs.
Subscriber Real-Time Activity Monitoring
This section provides details on Subscriber Real-Time Activity Monitoring.
This task involves:
•
Configuring the Collection Manager
•
Configuring the Cisco SCA Reporter
Configuring the Cisco SCE
Step 1
Configure Cisco SCE using the following configuration steps.
Step 2
Enter Configuration mode in SCE.
SCE2000#>configure
Step 3
Enter Interface Configuration mode in SCE.
SCE2000(config)#>in li 0
Step 4
Enable Online Flow Usage RDR.
SCE2000(config if)#>tunable GT_REP_OnlineFlowUsageRDR_Enable value true
Step 5
Set the monitor value to 1 to enable real time SUR or 4 to enable real time FUR.
Step 6
Exit from interface configuration mode.
SCE2000(config if)#>exit
Step 7
Exit from the configuration mode.
SCE2000(config)#>exit
Step 8
Copy Running configuration to startup configuration.
SCE2000#>copy running-config-application startup-config-application
Configuring the Collection Manager
Step 1
Confirm that the categorizer routes FURs to JDBC adapter. The tags property under JDBC Adapter should include the RDR tag 4042321927 in ~/cm/config/queue.conf.
Step 2
Uncomment the FUR table definition in the ~/cm/config/apps/scasbb/3.7.0/dbtables.xml file.
Step 3
(Optional) Confirm that the data reduction is enabled on RPT_FUR.
a.
If required, update ~/db_maint/dbperiodic.conf.
b.
Rerun ~/db_maint/create_periodic_del_procs.sh.
c.
Rerun ~/scripts/dbperiodic.py.
Configuring the Cisco SCA Reporter
Step 1
Confirm that the Subscriber Real Time Activity report enabled. If required, uncomment these in C:\Program Files\Cisco SCA\SCA Reporter 3.7.0\plugins\com.cisco.p3.apps.scasbbtemplates_3.7.0\templates\Subscriber Monitoring\group.xml:
<template file="Subscriber Real Time Activity.xml"/>
Offline Collection Manager Database Aggregation
The database scripts are provided in the distribution package under install-scripts/Aggregation.
Run these scripts on MySQL server. For example, by using the MySQL Query Browser.
General parameters
Create general parameters table.
Run the content of CREATE_PARAMETERS_TABLE.txt
xUR tables
Create aggregation stored procedures for xUR tables.
Run the contents of the following files:
•
P_RPT_LUR_AGGREGATE.txt
•
P_RPT_VLUR_AGGREGATE.txt
•
P_RPT_PUR_AGGREGATE.txt
•
P_RPT_MALUR_AGGREGATE.txt
BI RAG tables
Create aggregation stored procedures for BI RAG tables.
Run the contents of the following files:
•
P_RPT_TOP_HTTP_HOSTS_AGGREGATE.txt
•
P_RPT_TOP_HTTP_DOMAINS_AGGREGATE.txt
•
P_RPT_TOP_VIDEO_HOSTS_AGGREGATE.txt
•
P_RPT_TOP_ VIDEO_DOMAINS_AGGREGATE.txt
Events Scheduler
Create stored procedures scheduled runs.
Run the corresponding commands from all_events_create.txt for tables with aggregation enabled.
4 Obtaining Documentation and Submitting a Service Request
For information on obtaining documentation, submitting a service request, and gathering additional information, see the monthly What's New in Cisco Product Documentation, which also lists all new and revised Cisco technical documentation, at:
http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html
Subscribe to the What's New in Cisco Product Documentation as a Really Simple Syndication (RSS) feed and set content to be delivered directly to your desktop using a reader application. The RSS feeds are a free service and Cisco currently supports RSS Version 2.0.
Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1110R)
Any Internet Protocol (IP) addresses used in this document are not intended to be actual addresses. Any examples, command display output, and figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses in illustrative content is unintentional and coincidental.
© 2011 Cisco Systems, Inc. All rights reserved