Data Collection Manager
Cisco Data Collection Manager (DCM) is an efficient and reliable data collection agent that is embedded in managed devices, such as routers and switches. DCM works on a push model, which is based on a subscribe-and-notify data pattern, as opposed to the pull model, which is based on a request-and-response data pattern. The Data Collection Manager (DCM) supports advanced on-board data processing that includes baseline calculation, summary calculation, statistical distribution, and percentile computation.
Data Collection Manager and Bulkstat
The Data Collection Manager (DCM) and the bulkstat module are the vital units of a framework which enables the bulk collection mechanism to include multiple data sources and multiple data export mechanisms.
The Bulkstat client application is implemented using the DCM core services to retrieve data and export it to the user. The Bulkstat client provides the only available user interface for DCM access. The client also provides CLI access through a new set of configuration commands and MIB access through the CISCO-DATA-COLLECTION-MIB.
DCM provides data subscription service for different data sources (such as, SNMP MIB objects and show command outputs).It also provides data retrieval management and data filtering services. With DCM, one source can be allocated for periodically collecting all management data.
Bulkstat, is an application which will use DCM to provide the following:
-
Profiles and data-groups for different data-sources.
-
Data processing – Summary, Distribution, Percentile and Auto-baseline.
-
Data exports – File.
-
Calendar scheduling.
Benefits of DCM
DCM is very useful for Data Retrieval and Export and Performance Management solutions. This list includes all the benefits of DCM.
-
Data export and retrieval: The Data Collection Manager (DCM) provides data retrieval management to ensure that the data collection does not impact device resources. The DCM can export data in a file format using multiple export protocols such as FTP, TFTP, Secure copy protocol (SCP), and Secure File Transfer Protocol (SFTP). The DCM provides a query mechanism with which data can be selectively exported based on the configured time interval and other selection criteria. The DCM application also provides data filtering services and exports the filtered data. You can also set primary and secondary destinations for exporting the collected data in a raw or processed format. Snapshots of the collected data can be stored for later retrieval.
-
Performance Management: The Data Collection Manager (DCM) can be used to manage various aspects of performance management. It can collect data with a high granularity to help the Network Management Server (NMS) make dynamic traffic engineering decisions. DCM can also be used to collect resource variables that are important for effective capacity trend information, such as memory, queue depth, broadcast volume, buffer, Frame Relay congestion notification, and backplane utilization.
-
Troubleshooting: The streaming function of the DCM can be used for real-time troubleshooting.
-
SLA: A service level agreement (SLA) includes a what-if analysis for network changes and application changes, a trend for defined performance variables, exception management for defined capacity and performance variables, and QoS management. The DCM can be used to collect periodic data for reporting purposes.
Bulkstat
Two challenges that network providers usually face are data gathering and data analysis. Network providers need to gather large volumes of data to analyze the performance of the network and to have operational control over their network. Large service providers are strengthening their data gathering and analysis infrastructure. Traditionally, Simple Network Management Protocol (SNMP) agents are used to expose management data on managed systems. But, SNMP is not well suited for gathering large volumes of data, especially over short time intervals. For example, service providers charge customers depending on the network usage. Also this data must be available on customer request. Accounting applications based on SNMP polling models consume significant network bandwidth because they poll large volumes of data frequently. The SNMP protocol data unit (PDU) is a complex data type specific to SNMP and is expensive to process because the SNMP objects and tables must be sorted in a lexicographic order. All the entries in SNMP MIB tables are lexicographically ordered by their object identifiers, because there is an implied ordering in the MIB based on the order of the object identifiers. In such cases, the need to continuously poll large or bulk SNMP statistics can be avoided by using applications known as collectors to retrieve data.
The Bulkstat application is one such collector that uses the services of the Data Collection Manager (DCM) to provide the following functions:
-
Collecting SNMP MIB object values.
-
Processing the collected data to create summary, percentiles, and auto-baselined values.
-
Exporting collected data through simple file transfers.
-
Scheduling calendar events for data collection and export.
The Bulkstat application provides command-line access through a set of new configuration commands and exclusive MIB access through CISCO-DATA-COLLECTION-MIB to collect SNMP data.
You can configure Bulkstat for the following functions:
-
Specify the way Bulkstat retrieves bulk statistics.
-
Specify the time interval in seconds at which Bulkstat transfers data to receivers.
-
Specify the maximum size of the bulk statistics file.
-
Specify the context, instance, and period at which the system retrieves bulk statistics.
-
Configure file-related parameters.
-
Configure the interface type on which you want to collect statistics.
-
View the parameters that Bulkstat uses to collect statistics by using the show bulkstat commands.
Bulkstat Configuration Elements
The following list shows the elements that you can configure using the Bulkstat interface:
-
Data set
-
Instance set
-
Filter set
-
Data group
-
Process set
-
Data profile
-
Calendar Scheduling
Data Set
This section describes the data set elements that you can configure to collect Simple Network Management Protocol (SNMP) data and CLI data. Only objects having the same index elements can be grouped in a single object list.
The SNMP data set contains the following fields:
Name |
Description |
Configuration Status |
---|---|---|
Objects |
Specifies the object to be collected. Multiple objects can be configured to form a data set. The textual name of the object can be used for configuring an object. If the device does not recognize the textual name, the object identifier (OID) format can be used for configuring the name. |
Mandatory |
Object Alias |
Specifies the optional alias name that each object can have. |
Optional |
The CLI data set contains the following fields:
Name |
Description |
Configuration Status |
---|---|---|
CLI |
Specifies the CLI command for which the show output needs to be collected. More than one CLI can be specified in the same data set. |
Mandatory |
Filter Set
This section describes the filter configuration per object.
The filter set elements that you can configure to collect Simple Network Management Protocol (SNMP) data are described here. More than one filter of the same type can be added to the set.
Name |
Description |
Status |
---|---|---|
Object match |
Specifies the value to be used to match against the value retrieved for the object during collection. The value provided needs to match the type of the object. If there is an error in the type matching, the configuration is not accepted. More than one value can be specified for an object, and more than one object can have matching values. |
Optional |
Instance Set
This section specifies the instance set elements that you can configure to collect Simple Network Management Protocol (SNMP) data. More than one instance of the same type can be added to the set. Combinations of types of instance set elements are not supported.
The SNMP Instance set contains the following fields:
Name |
Description |
Configuration Status |
---|---|---|
Exact |
Specifies the instance for which the data should be collected. More than one instance can be specified, but only fully qualified instances should be specified. |
Optional |
Wildcard |
Specifies all instances for all objects under the object configured in the data set. |
Optional |
Range |
Specifies the start and end instances. All instances within the range, including the start and end, are collected, but only fully qualified instances should be specified. |
Optional |
Repetition |
Specifies the start of the repetition and the number of repetitions. All instances from the start until the number of repetitions within the subtree are collected. |
Optional |
Interface |
Specifies the interface instead of the index. The ifIndex assigned to the interface will be used as an index. This can be used for MIB objects indexed by ifindex. |
Optional |
Process Set
Data processing allows users to derive information from raw SNMP data, by calculating summaries and percentiles. Service providers rely on monitored SNMP data to alert network management systems (NMSs) of changing network conditions. By periodically monitoring the device data and comparing it against a set of thresholds, the network can automatically alert the operators, thereby allowing efficient operations.
-
Summary: You can enable summary processing on the collected object value and calculate minimum, maximum, and average values. A summary is calculated for only those objects that are marked as process capable in the data group and uses the absolute or delta value as per the object configuration.
-
Distribution: You can enable distribution processing on the collected object value by specifying the object type, minimum value, maximum value, and the number of buckets to distribute the value. Based on the configuration, counters are maintained per bucket and are incremented whenever the data falls into a bucket range.
-
Percentile: You can enable percentile processing on the collected object value. A percentile is calculated on every process interval expiry. Distribution configuration is mandatory to enable percentile processing. Percentile computation is done assuming that the distribution is normal.
-
Auto-baseline: You can enable baseline processing on the collected object value. The baseline internally uses all summary, distribution, and percentile calculations to provide baseline values. You can configure either baseline processing or other forms of processing, such as summary, distribution, and percentile calculations. The auto-baseline feature in DCM calculates the baseline values for variables of interest on the device and allows network management applications or network operators to retrieve the baseline values. The baseline values can be displayed in terms of percentiles or a median with standard deviation.
Data Group
This section describes the data group, which contains the data-group name, data-group type, data set, instance set, filter set, polling interval, SNMP context, and other processing options.
The Data Group elements are:
Name |
Description |
Configuration Status |
||
---|---|---|---|---|
Data |
Specifies any one of the data types as defined in the topic Data Set. |
Mandatory |
||
Instance |
Specifies any one of the instance types as defined in the topic Instance Set. |
Optional, if not specified.Default behavior of the instance set is wildcard. Only applicable for SNMP. |
||
Filter |
Specifies any one of the filter types as defined in the topic Filter Set. |
Optional, if not specified. Only applicable for SNMP. |
||
Polling Interval |
Specifies the collection periodic interval in seconds. In case of recurring collection, the data is collected at the expiration of the collection interval until the collection is stopped. |
Optional |
||
Context |
Specifies the management context from which to obtain data for this data group. |
Optional |
||
Process Summary |
Enables summary processing of the data marked to be processed in the corresponding data-set configuration. |
Optional |
||
Process Distribution |
Enables distribution processing of the data marked to be processed in the corresponding data-set configuration. |
Optional |
||
Process Percentile |
Enables percentile processing of the data marked to be processed in the corresponding data-set configuration. |
Optional |
||
Process Auto-baseline |
Enables auto-baselining processing of the data marked to be processed in the corresponding data-set configuration. If auto-baseline process is enabled, the other processes, such as summary, distribution, and percentile configurations, if done previously, are removed because auto-baseline process uses these functionalities internally.
|
Optional |
||
Discard raw |
Specifies whether to store raw data. If data is processed, the user can choose to store only process data by setting the option. |
Optional |
Data Profile
This section describes the data profile that is used to group multiple data groups. This is done to simplify the configuration and to aggregate data of similar nature. A data profile can have multiple data groups. A data group can have constraints in the data specified in the element. If two sets of data need to be written to the same file, the respective data groups should be linked as part of a single profile.
The Data Profile has these fields:
Name |
Description |
Status |
---|---|---|
Data groups |
Specifies the data group to be linked to this profile. Multiple data groups can be linked to a single profile. |
Mandatory before activating a profile |
Transfer Interval |
Specifies the transfer periodic interval in seconds. In case of recurring transfer, the data is transferred when the transfer interval expires. |
Optional |
Process Interval |
Specifies the process periodic interval in seconds. The data is processed during every collection interval as soon as it is collected. When the process interval expires, the processed data is written into a file and transferred. |
Optional |
Primary URL |
Specifies the URL of the primary management station. The files containing the collected data are transferred to this URL when the transfer interval expires. |
Mandatory |
Secondary URL |
Specifies the URL of the secondary management station to be used in case the transfer to the primary management station fails. |
Optional |
Schema |
Specifies the file data format. The schema ASCII option is supported. |
Optional |
Retry |
Specifies the number of times that the transfer is retried in case of transfer failures to both primary and secondary management stations. This command has an effect only if the retain command is configured in the profile. The retry interval is computed by dividing the retention time by the number of retries. For example, if the file is retained for 60 minutes and the retry is 6 times, the transfer is attempted every 10 minutes, until the transfer succeeds or the file is removed. |
Optional |
Buffer-size |
Specifies the maximum size to which the file containing the collected data can grow. When it reaches the limit, the file is closed and the transfer is attempted based on the transfer configuration associated with the data group or profile. |
Optional |
Retention Memory |
Specifies the time, in seconds, to retain the file in the memory. |
Optional |
Retention USB |
Specifies the time, in seconds, to retain the file in the USB. This option is available only if the device supports the USB drive. |
Optional |
Calendar Scheduling
The Bulkstat application allows you to schedule each subscription for collection. A subscription can be scheduled for one-time collection or periodic collection. A periodic subscription can be repeated infinitely or for a specified number of repetitions. A timer is instantiated for every activated subscription.
The calendar scheduling elements are:
Name |
Description |
Configuration Status |
---|---|---|
One shot |
Specifies that the data is collected for a specified collection interval. |
Optional |
Recurring |
Specifies that the data is collected regularly at the specified time, day, month, and for a specified collection interval. |
Optional |
File Data Export
The file data export feature on the Data Collection Manager (DCM) exports the collected data based on the transfer configurations. Data can be exported in various formats, and Bulkstat files are one such format to collect data. The format in which the data is inserted into the file conforms to the schema-Ascii format described in CISCO-DATA-COLLECTION-MIB and CISCO-BULK-FILE-MIB. The data sequence in which the data is stored is determined based on the sequence in which the data is received.
The Cisco File Transfer module is responsible for transferring the files as per the transfer configuration. A file can be retained in the device whether the transfer was a success or a failure.
Configuring an SNMP Bulkstat Data Set
The first step in configuring the Simple Network Management Protocol (SNMP) periodic data collection and transfer mechanism is to configure one or more data sets. A data set is used to group objects of similar types, based on the data source. The data set is defined outside of the data group. This external definition gives the user the flexibility to use the same data set across multiple data groups and to collect the output for different instances and different contexts.
All objects in an SNMP data set must be indexed by the same MIB index. However, the objects in the data set must not belong to the same MIB or the MIB table.
Perform this task to configure the SNMP Bulkstat data set.
SUMMARY STEPS
- configure
- bulkstat data data-set -name type snmp
- object oid [ alias alias-name ]
DETAILED STEPS
Command or Action | Purpose | |||
---|---|---|---|---|
Step 1 |
configure Example:
|
Enters global configuration mode. |
||
Step 2 |
bulkstat data data-set -name type snmp Example:
|
Defines an SNMP Bulkstat data set and enters SNMP bulk statistics data set configuration mode. The creation of an SNMP Bulkstat data set creates a row in the cdcDGBaseObjectEntry table in the SNMP MIB. |
||
Step 3 |
object oid [ alias alias-name ] Example:
|
Adds a MIB object to the SNMP Bulkstat data set. If the object is already present in the data set, this command replaces the old object configuration with the new configuration.
|
Configuring an SNMP Bulkstat Filter Set
The Simple Network Management Protocol (SNMP) filter set specifies the filter configuration for every SNMP object.
Perform this task to configure the SNMP Bulkstat filter set.
SUMMARY STEPS
- configure
- bulkstat filter filter-set -name
- match object-name { eq line | start line | not { eq line | start line } }
DETAILED STEPS
Command or Action | Purpose | |||
---|---|---|---|---|
Step 1 |
configure Example:
|
Enters global configuration mode. |
||
Step 2 |
bulkstat filter filter-set -name Example:
|
Defines an SNMP Bulkstat filter set and enters SNMP bulk statistics filter set configuration mode. |
||
Step 3 |
match object-name { eq line | start line | not { eq line | start line } } Example:
|
(Optional) Specifies a value to be used to match against the value retrieved for the object during collection.
|
Configuring an SNMP Bulkstat Instance Set
The Simple Network Management Protocol (SNMP) instance set specifies the instances for which the data should be collected. Each subscription can collect different entries for specified objects based on the instance configuration. While more than one instance of the same type can be added to the instance set, a combination of different types is not supported.
Perform this task to configure the SNMP Bulkstat instance set.
SUMMARY STEPS
- configure
- bulkstat instance instance-set -name type snmp
- exact oid oid
- exact interface interface-id
- wildcard
- wildcard oid oid
- wildcard interface interface-id
- repetition oid oid max value
- range start oid end oid
DETAILED STEPS
Command or Action | Purpose | |||
---|---|---|---|---|
Step 1 |
configure Example:
|
Enters global configuration mode. |
||
Step 2 |
bulkstat instance instance-set -name type snmp Example:
|
Defines an SNMP Bulkstat instance set and enters SNMP Bulkstat instance set configuration mode. The creation of an SNMP Bulkstat instance set creates a row in the cdcDGInstanceEntry table in the SNMP MIB.
|
||
Step 3 |
exact oid oid Example:
|
(Optional) Indicates that the specified instance, when appended to the object list, is the complete OID. |
||
Step 4 |
exact interface interface-id Example:
|
(Optional) Specifies an interface name and number, for example interface Ethernet 0, instead of specifying the ifIndex OID for the interface. |
||
Step 5 |
wildcard Example:
|
(Optional) Specifies whether an object used for evaluating an expression should be made a wildcard during an event configuration. |
||
Step 6 |
wildcard oid oid Example:
|
(Optional) Indicates that all subindices of the specified OID belong to this schema. |
||
Step 7 |
wildcard interface interface-id Example:
|
(Optional) Specifies an interface name and number, for example interface Ethernet 0, instead of specifying the ifIndex OID for the interface. |
||
Step 8 |
repetition oid oid max value Example:
|
(Optional) Configures data collection to repeat get-next for the maximum number of instances starting from the specified oid instance. |
||
Step 9 |
range start oid end oid Example:
|
(Optional) Configures a range of instances for which the data is collected. |
Configuring a Bulkstat Data Group
The Bulkstat data group element is used to group the data set, filter set, and instance set and also to specify the processing options.
Perform this task to configure the Bulkstat data group.
SUMMARY STEPS
- configure
- bulkstat data-gorup data-group-name
- collect type { { command | expression } date date-set-name filter filter-set-name | snmp { data data-set-name instance instance-set-name filter filter-set-name } }
- context context-name
- interval polling polling-interval
- discard
- process
DETAILED STEPS
Command or Action | Purpose | |
---|---|---|
Step 1 |
configure Example:
|
Enters global configuration mode. |
Step 2 |
bulkstat data-gorup data-group-name Example:
|
Defines a Bulkstat data group and enters Bulkstat data group configuration mode. The creation of a Simple Network Management Protocol (SNMP) Bulkstat data group creates a row in the cdcDGEntry table in the SNMP MIB. |
Step 3 |
collect type { { command | expression } date date-set-name filter filter-set-name | snmp { data data-set-name instance instance-set-name filter filter-set-name } } Example:
|
Specifies the collection type to collect data from different sources for this data group. |
Step 4 |
context context-name Example:
|
Specifies the management context from which to obtain data for this data group. |
Step 5 |
interval polling polling-interval Example:
|
Specifies the collection periodic interval in seconds. In case of recurring collection, the data is collected at the expiration of the collection interval until the collection is stopped. |
Step 6 |
discard Example:
|
Specifies whether to discard the raw data. |
Step 7 |
process Example:
|
Configures process-related parameters for a data group. |
Configuring a Bulkstat Profile
Perform this task to configure the Bulkstat Profile.
The profile element is used to group multiple data groups. This grouping simplifies the configuration and aggregates data of a similar nature. If two sets of data need to be written to the same file, the respective data groups should be linked as part of a single profile.
SUMMARY STEPS
- configure
- bulkstat profile profile-name
- data-group data-group name
- interval transfer { process | raw} seconds
- file-format schema ASCII
- file retain { disk url | memory seconds }
- file size bytes
- file transfer { retry number | url { primary url | secondary url }}
- enable
DETAILED STEPS
Command or Action | Purpose | |
---|---|---|
Step 1 |
configure Example:
|
Enters global configuration mode. |
Step 2 |
bulkstat profile profile-name Example:
|
|
Step 3 |
data-group data-group name Example:
|
|
Step 4 |
interval transfer { process | raw} seconds Example:
|
Specifies the transfer periodic interval in seconds. In case of recurring transfer, the data is transferred at the expiration of the transfer interval until the transfer is stopped. |
Step 5 |
file-format schema ASCII Example:
|
Configures the file-related parameter for a profile. Specifies the file data format in ASCII. |
Step 6 |
file retain { disk url | memory seconds } Example:
|
Configures the file-related parameter for a profile.
|
Step 7 |
file size bytes Example:
|
Configures the file-related size parameter for a profile. size - Specifies the maximum buffer size in bytes. When the limit is reached, the file is closed and transfer is attempted based on the transfer configuration associated with the data group or the profile. |
Step 8 |
file transfer { retry number | url { primary url | secondary url }} Example:
|
Configures the file-related transfer parameter for a profile.
|
Step 9 |
enable Example:
|
Enables the profile for collection and transfer. |
Configuring Bulkstat Calendar Scheduling
SUMMARY STEPS
- configure
- bulkstat schedule schedule at time-detail { oneshot | recurring }
- profile profile-name start { oneshot | recurring number }
- profile profile-name stop
DETAILED STEPS
Command or Action | Purpose | |
---|---|---|
Step 1 |
configure Example:
|
Enters global configuration mode. |
Step 2 |
bulkstat schedule schedule at time-detail { oneshot | recurring } Example:
|
Defines the Bulkstat calendar scheduler set and enters Bulkstat event scheduler configuration mode. For the time-detail option, enter the details of the time as prompted. First the time in the 24-hour clock format, followed by the month and then the date. |
Step 3 |
profile profile-name start { oneshot | recurring number } Example:
|
Creates a profile and sets the condition to enable the profile for a one-time event or enables the profile for multiple events. |
Step 4 |
profile profile-name stop Example:
|
Disables the profile. |
Configuration Examples and Usecase Scenarios
The usecase scenarios with examples are discussed here.
Usecase-1: Collecting MIB Statistics
Goal: To collect IF MIB Statistics
Procedure |
Example |
---|---|
Step1: Identifying the inputs and other parameters |
MIB Objects of interest:
Export Parameters:
|
Step2: Configuring the Data set if-mib For detailed procedure: Configuring an SNMP Bulkstat Data Set |
|
Step3: Configuring the Instance set if-mib For detailed procedure: Configuring an SNMP Bulkstat Instance Set |
|
Step4: Configuring Data Group if-group For detailed procedure: Configuring a Bulkstat Data Group |
|
Step5: Configuring Profile snmp_profile For detailed procedure: Configuring a Bulkstat Profile |
|
Note |
Step2 and Step3 can be interchanged. |
Usecase-2: Using Filters
Goal: To collect gigabit ethernet interface statistics (using filters)
Procedure |
Example |
---|---|
Step1: Identifying the inputs and other parameters |
MIB Objects of interest:
Export Parameters:
|
Step2: Configuring the Data set if-mib For detailed procedure: Configuring an SNMP Bulkstat Data Set |
|
Step3: Configuring the Instance set if-mib For detailed procedure: Configuring an SNMP Bulkstat Instance Set |
|
Step4: Configuring the Filter set if-mib For detailed procedure: Configuring an SNMP Bulkstat Filter Set |
Setting the filter (in this case, it is - gigabit ethernet interface)
|
Step5: Configuring Data Group if-group For detailed procedure: Configuring a Bulkstat Data Group |
|
Step6: Configuring Profile snmp_profile For detailed procedure: Configuring a Bulkstat Profile |
|
Note |
Step2, Step3 and Step4 can interchanged. |
Usecase-3: Collecting CLI output in XML format
Goal: To collect show cli output in XML format
Procedure |
Example |
---|---|
Step1: Identifying the inputs and other parameters |
CLI of interest: add cmd show operational AAA xml Export Parameters:
|
Step2: Configuring the Data set process For detailed procedure: Configuring an SNMP Bulkstat Data Set |
|
Step3: Configuring Data Group cli-group For detailed procedure: Configuring a Bulkstat Data Group |
|
Step4: Configuring Profile cli_profile For detailed procedure: Configuring a Bulkstat Profile |
|