The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This chapter describes several of problem scenarios that you may encounter when using the solution. The problem scenarios and solutions are described under the following sections:
•Inaccurate, Unavailable, or Missing Reports Information for a Specific CMTS Interface
•SCE is Congested, But Connected CMTSs are Not Congested
To troubleshoot the cause of bad service to subscribers, consider the following:
•Congestion in the backbone network may cause congestion in subscriber traffic.
•Problem with the solution, such as assigning the incorrect policy to the subscriber, or an incorrect interface association, may exist.
To make sure the problem is not with the SCE, perform the following procedures:
Step 1 Verify that the subscriber manager-SCE subscriber data is synchronized.
In the SCE, use the show interface linecard 0 subscriber name <sub MAC> command to verify that the upvlinkId and downVlinkId values are the same as in the p3subs --show -s <sub MAC> subscriber manager CLU output and that the subscriber package ID is as expected.
The following example shows the output of the SCE CLI:
SCE2000#>show interface LineCard 0 subscriber name lynn_jones
Subscriber 'lynn_jones' manager: SM
Subscriber 'lynn_jones' properties:
downVlinkId=10
monitor=0
new_classification_policy=0
packageId=0
QpLimit[0..17]=0*17,8
QpSet[0..17]=0*17,1
upVlinkId=10
.
.
.
The following example shows the output of the subscriber manager CLU:
>p3subs --show -s lynn_jones
Name: lynn_jones
Domain: subscribers
Mappings:
IP: 5.101.5.129/32
Properties:
downVlinkId=10 Name=dev0_9_if19-down
upVlinkId=10 Name=dev0_9_if19-upstream0
Custom Properties:
giaddr=5.101.254.105
Command terminated successfully
Step 2 Verify that the subscriber manager-SCE virtual links data is synchronized.
Compare the output of the SCE show interface linecard 0 virtual-links different-from-template command with the subscriber manager CLU p3vlink --show-vlink-data --vlink-name <UpVlinkId/DownVlinkId Name> to make sure that the PIR configuration is correct.
Note The SCE PIR value is in kilobits. You can calculate the VLM value by performing a multiplication with the relevant CMTS device factor as defined in the vlink.cfg configuration file. The CMTS device factor value appears in the output of the subscriber manager CLU p3vlink --show-device -d <device-name>.
The following example shows the output of the SCE CLI:
SCE2000#> show interface LineCard 0 virtual-links different-from-template
Virtual Link enabled
upstream Virtual-Link:
virtual index=1, name=virtual link 1
channel index=1, name=virtual channel 1, pir=99999, cir=99, al=9, agc index=3
application index=1, name=appGC-1, pir=8000000, cir=0, al=5, agc index=2
virtual index=2, name=virtual link 2
channel index=2, name=virtual channel 2, pir=99999, cir=99, al=9, agc index=7
application index=1, name=appGC-2, pir=8000000, cir=0, al=5, agc index=6
downstream Virtual-Link:
virtual index=3, name=virtual link 3
channel index=3, name=virtual channel 3, pir=99999, cir=99, al=9, agc index=15
application index=1, name=appGC-3, pir=8000000, cir=0, al=5, agc index=12
application index=2, name=appGC1-3, pir=8000000, cir=0, al=5, agc index=13
application index=3, name=appGC2-3, pir=8000000, cir=0, al=5, agc index=14
The following example shows the output of the p3vlink --show-vlink-data subscriber manager CLU:
p3vlink --show-vlink-data --vlink-name=device_Cmts0/0-downstream1
VLink Name: device_Cmts0/0-downstream1
VLink Id: 1
Direction: downstream
SCE Name: sce0
Device Name: device
PIR: 200000000
Channels related to VLink
<name>-L, index <index>, PIR <value>, CIR <value>
<name>-W, index <index>, PIR <value>, CIR <value>
Related upstream virtual links —Lists all upstream interface related to the same MAC layer as the selected downstream interface.
device_Cmts0/0-upstream0
device_Cmts0/0-upstream1
device_Cmts0/0-upstream2
device_Cmts0/0-upstream3
Command terminated successfully
The following example shows the output of the p3vlink --show-device subscriber manager CLU:
p3vlink --show-device -d CMTS1 --detail
Name: CMTS1
Host Name: Paris
IP: 192.0.2.10
Type: Static
SCE Related: sce0
Upstream factor: 95
Downstream factor: 95
Last success Query: Thu Jun 19 17:54:48 IDT 2008
Last Query Attempt: Thu Jun 19 17:54:48 IDT 2008
Last Query Status: Completed
Sync state with SCE: done
Sync state with CM: done
Giaddr List: 127.0.0.1
Upstream Global Controllers: None
Downstream Global Controllers: <GC Name>=<GC Value>,<GC Name>=<GC Value>...
isLogAll: true
Num of up interfaces: 6
Num of down interfaces: 2
VLink Information:
1) Name: CMTS1_Cmts0/0-upstream2, Vlink Id: 1, Direction UP, PIR 5000 kbps.
2) Name: CMTS1_Cmts0/0-downstream1, Vlink Id:1, Direction DOWN, PIR 10000 kbps
Channel Name: <vlink Name>-W, index <value>, PIR <value> kpbs, CIR <value> kpbs
Channel Name: <vlink Name>-L<channel index>, index <value>, PIR <value> kpbs, CIR <value> kpbs
3) Name: CMTS1_Cmts0/0-upstream3, Vlink Id:2, Direction UP, PIR 10000 kbps.
4) Name: CMTS1_Cmts1/0-downstream1, Vlink Id:2, Direction DOWN, PIR 20000 kbps.
5) Name: CMTS1_Cmts0/0-upstream1, Vlink Id:3, Direction UP, PIR 10000 kbps.
6) Name: CMTS1_Cmts1/0-upstream2, Vlink Id:4, Direction UP, PIR 20000 kbps.
7) Name: CMTS1_Cmts1/0-upstream3, Vlink Id:5, Direction UP, PIR 20000 kbps.
8) Name: CMTS1_Cmts1/0-upstream1, Vlink Id:6, Direction UP, PIR 20000 kbps.
Command terminated successfully
>
To fix this problem:
a. Monitor all CMTS device interfaces using the subscriber manager CLU p3vlink --show-device -d NAME [--detail].
b. Verify that in the output of the CLU Sync state with SCE is done.
If Sync state with SCE is not done:
–View the connection state between the subscriber manager and the SCE using the subscriber manager CLU p3net --show -n SCE_NAME.
–Synchronize the SCE with the CMTS device configuration using the subscriber manager CLU p3vlink --resync -n SCE_NAME.
Step 3 Verify that the VLM-CMTS data is synchronized.
Compare the virtual link names from the subscriber manager CLU p3subs --show -s <sub MAC> output with the CMTS configuration to verify that the subscriber is assigned to the correct CMTS and CMTS interface.
Note The virtual link name structure is <CTMS_NAME>_<INTERFACE_DESCRIPTION> and is obtained from the configuration file and the SNMP ifTable data.
Make sure that the PIR value is the same as the CMTS interface speed (as taken from the SNMP ifTable data) by comparing the virtual link PIR values from the subscriber manager CLU p3vlink --show-vlink-data --vlink-name <UpVlinkId/DownVlinkId Name> to the CMTS configuration.
To fix this problem:
a. Use p3vlink --show-device -d <DEVICE_NAME> to verify that:
–The monitor period is not 0
–The last query completed successfully. Check the user-log for reasons for failure.
b. Use p3vlink --start-query -d <DEVICE_NAME> to force a start query on a specific CMTS device.
Step 1 Obtain the VLM name for a specific interface using the p3subs --show -s <sub MAC> command, or build the interface name as <CTMS_NAME>_<INTERFACE_DESCRIPTION>.
Step 2 Retrieve the subscribers list related to a specific interface and compare it to the CMTS data using the p3vlink --show-subs --vlink-name= <UpVlinkId/DownVlinkId Name> command:
>p3vlink --show-subs --vlink-name test1_Cmts0/0-upstream2
Subscribers related to device: test1 vlink-id: 5, giaddr: 10.78.233.149, direction UP
010101010101
1 subscriber was found
Command terminated successfully
>
The SCA Reporter can generate per interface consumption reports that can be used to monitor the solution. The VLM updates the reporter with the interface ID to name data and the SCE sends the raw data with the interface ID (virtual link).
A problem can stem from the VLM update or from the SCE RDRs.
Two possible solutions include:
•Verify that the SCE sends virtual link update RDRs (VLUR) to the collection manager. For information, see Cisco SCE8000 10GBE Software Configuration Guide or Cisco SCE8000 GBE Software Configuration Guide.
•Verify that the VLM updates the collection manager.
Step 1 Verify the connectivity between the subscriber manager and the collection manager.
View the configured network elements and verify that the collection manager exists using the subscriber manager CLU p3net --show-all --detail command.
View the subscriber manager-collection manager connection properties and state using the p3net --show -n CM_NAME command, and then verify that the SCE list property value contains the SCE to which the CMTS is connected.
To fix this problem:
a. Check that the subscriber manager configuration file contains the relevant collection manager information:
–Check that a CM Section exists and points to the SCE section
–Validate the collection manager IP and port
–Check that the SCE is related to the collection manager.
b. See the "Information About Communication Failures" section in the "Subscriber Manager Overview" chapter of Cisco Service Control Management Suite Subscriber Manager User Guide.
Step 2 Verify that the VLM and collection manager data is synchronized.
Verify that the synchronization state with collection manager is set to done using the CLU p3vlink --show-device -d <CMTS_NAME> command. To view a list of configured virtual links, use the collection manager CLU update_vlinks.sh --sce=<SCE_IP> --show command.
To fix this problem:
a. Force the VLM and the collection manager to synchronize using the p3vlink --resync -n <CM_NAME> command.
b. For manual configuration, see Cisco Service Control Management Suite Collection Manager User Guide to add virtual links using the CLU update_vlinks.sh --sce=<SCE_IP> --file=vlinks.csv.
The following example shows the output of the p3net --show -n command:
>p3net --show -n cm0
Network Element Information:
============================
Name: cm0
Host: 10.56.197.231
Ip: 10.56.197.231
Port: 14375
Status: Connection ready
Type: Collection Manager
SCE List: sce0
Synchronization Status: Not-done(100% failures)
Redundancy Status: Standalone
Quarantine Status: ok
Command terminated successfully
The following example shows the output of the p3vlink --show-device -d command:
>p3vlink --show-device -d test1
Device Name: test1
Host Name: singleSimpleDevices
IP: 10.78.233.149
Type: Static
SCE Related: sce1
Upstream factor: 95
Downstream factor: 95
Last success Query: Fri Feb 19 14:29:11 IST 2010
Last Query Attempt: Fri Feb 19 14:29:11 IST 2010
Last Query Status: Completed
Sync state with SCE: Not-done
Sync state with CM: N/A
Giaddr List: 10.78.233.149;
Upstream Global Controllers: None
Downstream Global Controllers: None
isLogAll: true
Num of up interfaces: 8
Num of down interfaces: 2
Command terminated successfully
The following example shows the output of the update_vlinks.sh script:
./update_vlinks.sh --sce=10.56.197.232 --show
.
.
.
TIME_STAMP| SE_IP| VLINK_ID| VLINK_DIRECTION| VLINK_NAME|
----------+------------------+------------------+------------------+---------------------- --+
.
.
.
2008-12-17| 10.56.197.232| 10| 0| dev0_9_if19-upstream0|
2008-12-17| 10.56.197.232| 10| 1| dev0_9_if19-down|
.
.
.
If the virtual link assignment is incorrect or false subscriber login operations are not handled by the correct virtual link controller, virtual links can become congested. To resolve this issue, you must confirm that there are no subscribers associated with the virtual link and also verify that the distribution of subscribers to the virtual link is as expected.
To verify that the distribution of subscribers to the virtual link is as expected, see the "Verifying that the Distribution of Subscribers to the Virtual Link is Correct" section.
Step 1 Use the SCE CLI show interface LineCard 0 subscriber property <upVlinkId|downVlinkId> equals 0 to make sure that there are no subscribers associated with the default virtual link.
Subscribers in the SCE can have a virtual link ID set to 0 in several cases:
•If the giaddr list is learned automatically, a few subscribers may have this policy if they performed the first login from this giaddr. In this case, compare the results with the subscriber manager database.
•If a specific interface or CMTS is removed from the VLM, at some point all the subscribers associated with this interface or CMTS are set to use the default virtual link.
•If no match exists in the DHCP Sniffer LEG mapping table and the LEG is configured to log in a subscriber with the default virtual link.
The following example shows the output of the show interface LineCard 0 subscriber property upVlinkId equals 0 command:
SCE2000#>show interface LineCard 0 subscriber property upVlinkId equals 0
N/A
000004650001
000004650002
000004650003
000004650004
000004650005
000004650006
000004650007
000004650008
000004650009
00000465000A
00000465006F
000004650070
000004650071
000004650072
000004650073
000004650074
000004650075
000004650076
000004650077
000004650078
0000046500DD
0000046500DE
0000046500DF
The following example shows the output of the p3dhcpsniff --show-policy --policy=upVlinkId --detail command:
>p3dhcpsniff --show-policy --policy=upVlinkId --detail
Policy Name: upVlinkId
=======================
separator:_
use default: false
default value: 0
allow no package: true
concat option:giaddr,
concat option:82:1,
concat option type: integer,binary
log success: true
log default success: false
Number of mappings: 400
5.101.254.100_00010000=9
5.101.254.100_00010003=10
5.101.254.100_80010000=9
5.101.254.100_80010003=10
5.101.254.101_00010000=9
5.101.254.101_00010003=10
5.101.254.101_80010000=9
5.101.254.101_80010003=10
5.101.254.102_00010000=9
5.101.254.102_00010003=10
5.101.254.102_80010000=9
5.101.254.102_80010003=10
5.101.254.103_00010000=9
5.101.254.103_00010003=10
5.101.254.103_80010000=9
5.101.254.103_80010003=10
5.101.254.104_00010000=9
5.101.254.104_00010003=10
5.101.254.104_80010000=9
5.101.254.104_80010003=10
5.101.254.105_00010000=9
5.101.254.105_00010003=10
5.101.254.105_80010000=9
5.101.254.105_80010003=10
5.101.254.106_00010000=9
5.101.254.106_00010003=10
5.101.254.106_80010000=9
The messages that are written to the userlog are categorized based on their severity as:
The preconditions of the messages are described as follows:
•None—Messages are sent irrespective of the value set in the config file.
•General.log_all—Messages are sent only if log_all flag in the general section of the vlink.cfg file is set to true.
•Device.log_all—Messages are sent only if log_all flag in the device section is set to true.
Table 6-1 lists the error messages that can be written to the userlog.
Table 6-3 lists the warning messages that can be written to the userlog.
Table 6-3 lists the information messages that can be written to the userlog.