Troubleshooting Bouncing IMA Links
Available Languages
Contents
Introduction
Inverse multiplexing over ATM (IMA) defines a sublayer-1 protocol for building a virtual link that consists of multiple physical T1 or E1 links. The IMA protocol handles link failure and automatic link recovery, and it also adds and deletes links while keeping the IMA group in service.
This document provides troubleshooting steps for you to follow when your router reports up or down link changes to a member interface in an IMA group or when the group interface bounces. Physical T1 interfaces are removed from a group under these conditions:
-
Exceeding the differential delay.
-
Experiencing a physical layer alarm or error state.
Prerequisites
Requirements
Readers of this document should have knowledge of these topics:
Components Used
This document is not restricted to specific software and hardware versions.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Conventions
For more information on document conventions, refer to the Cisco Technical Tips Conventions.
Decoding the Active Link Bitmap
Before understanding the two reasons for physical T1 lines to be removed from an IMA group, it is important to know how to determine which link was removed.
In this sample output, you can see the IMA group repeatedly reporting a change in the number of active links in the group.
Apr 13 20:45:47.196 CDT: %IMA-5-ACTIVE_LINK_CHANGE: IMA Group ATM2/IMA0 now has 5 active links, active link bitmap is 0x37. APR 13 20:45:47.964 CDT: %IMA-5-ACTIVE_LINK_CHANGE: IMA Group ATM2/IMA0 now has 6 active links, active link bitmap is 0x3F. APR 13 20:45:51.184 CDT: %IMA-5-ACTIVE_LINK_CHANGE: IMA Group ATM2/IMA0 now has 5 active links, active link bitmap is 0x37. APR 13 20:45:51.440 CDT: %IMA-5-ACTIVE_LINK_CHANGE: IMA Group ATM2/IMA0 now has 6 active links, active link bitmap is 0x3F. APR 13 20:45:55.528 CDT: %IMA-5-ACTIVE_LINK_CHANGE: IMA Group ATM2/IMA0 now has 5 active links, active link bitmap is 0x37.
The important thing to note about this output is the active link bitmap. The rest of this section discusses how to decode this bitmap. You can see that the bitmap consists of two hexadecimal characters or eight bits. Read these bits from right to left, with physical T1 port 0 at bit position 0 and physical T1 port 7 at bit position 7.
With the above output as an example and using these tables, you can see that when the active link bitmap changes from 0x37 to 0x3F, physical port 3 drops from the IMA group.
| Decode for 0x3F | ||||||||
|---|---|---|---|---|---|---|---|---|
| 2x Value | 8 | 4 | 2 | 1 | 8 | 4 | 2 | 1 |
| Binary Value | 0 | 0 | 1 | 1 | 1 | 1 | 1 | 1 |
| T1 Port Number | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |
| Decode for 0x37 | ||||||||
|---|---|---|---|---|---|---|---|---|
| 2x Value | 8 | 4 | 2 | 1 | 8 | 4 | 2 | 1 |
| Binary Value | 0 | 0 | 1 | 1 | 0 | 1 | 1 | 1 |
| T1 Port Number | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |
Note: Because repetitious bursts of errors affect the data path if links are brought back in too rapidly, the IMA specification suggests that vendors implement “de-bouncing” techniques to regulate T1 link addition and removal.
Physical Layer Alarm or Incrementing Errors
Issue the show controllers atm command on the physical T1 to determine the line status.
router# show controllers atm0/2 Interface ATM0/2 is administratively down Hardware is ATM T1 !--- Output suppressed. Link 2 Framer Info: framing is ESF, line code is B8ZS, fdl is ANSI cable-length is long, Rcv gain is 26db and Tx gain is 0db, clock src is line, payload-scrambling is disabled, no loopback line status is 0x1064; or TX RAI, Rx LOF, Rx LOS, Rx LCD. port is active, link is unavailable 0 idle rx, 0 correctable hec rx, 0 uncorrectable hec rx 0 cells rx, 599708004 cells TX, 0 rx fifo overrun. Link (2):DS1 MIB DATA: Data in current interval (518 seconds elapsed): 0 Line Code Violations, 0 Path Code Violations 0 Slip Secs, 518 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins 0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 519 Unavail Secs Total Data (last 24 hours) 0 Line Code Violations, 0 Path Code Violations, 0 Slip Secs, 86400 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins, 0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 86400 Unavail Secs
The line status bitmap comes from the dsx1LineStatus section of RFC 1406 
, the digital signal level 1 (DS-1) management information base (MIB). This contains loopback, failure, received alarm, and transmitted alarm information. The dsx1LineStatus is a bitmap represented as a sum, and thus can simultaneously represent multiple failures (alarms) and a loopback state. These are the various bit positions:
| Bit Value | Meaning | Definition |
|---|---|---|
| 1 | dsx1NoAlarm | No alarm present. |
| 2 | dsx1RcvFarEndLOF | Far end loss of frame (LOF); also known as a Yellow Alarm. |
| 4 | dsx1XmtFarEndLOF | Near end sending LOF indication. |
| 8 | dsx1RcvAIS | Far end sending alarm indication signal (AIS). |
| 16 | dsx1XmtAIS | Near end sending AIS. |
| 32 | dsx1LossOfFrame | Near end LOF; also known as a Red Alarm. |
| 64 | dsx1LossOfSignal | Near end loss of signal (LOS). |
| 128 | dsx1LoopbackState | Near end is looped. |
| 256 | dsx1T16AIS | E1 TS16 AIS. |
| 512 | dsx1RcvFarEndLOMF | Far end sending TS16 loss of multiframe (LOMF). |
| 1024 | dsx1XmtFarEndLOMF | Near end sending TS16 LOMF. |
| 2048 | dsx1RcvTestCode | Near end detects a test code. |
| 4096 | dsx1OtherFailure | Any line status that is not defined in this list. |
With the IMA network module, Cisco uses the value 4096 to indicate loss of cell delineation.
Note: ATM cell delineation and cell misinsertion problems can affect the IMA frame and lead to problems with an IMA interface. In these cases, the router typically does not report physical layer errors or alarms for the physical T1 interfaces.
Note: The clock source on an IMA link changes from line to internal when an AIS or an LOS is detected.
When troubleshooting IMA link drops, issue the show log command to check the router’s log for link up or down messages, as shown in this sample output:
APR 2 13:57:18: %LINK-3-UPDOWN: Interface ATM1/1, changed state to down APR 2 13:57:18: %LINK-3-UPDOWN: Interface ATM1/3, changed state to up APR 2 13:57:18: %LINK-3-UPDOWN: Interface ATM1/2, changed state to up
Cisco offers several T1 and E1 troubleshooting documents to assist you in resolving a line status message other than no alarm present.
E1 Troubleshooting Documents
T1 Troubleshooting Documents
Exceeding Differential Delay
In addition to rebuilding the original ATM cell stream, a receiving IMA interface compensates for delays in timing and in the arrival of adjacent ATM cells. A receiving interface detects and rejects physical links with delay greater than the provisioned maximum differential delay tolerance. This tolerance refers to the differences in the arrival of adjacent cells; that is, cells may be delayed more on one link than on a second link. If a link delay exceeds the specified maximum, the link is dropped. Otherwise, the IMA feature—while multiplexing and demultiplexing—adjusts for differences in delays so that all links in a group are aligned.
A receiving interface uses IMA control protocol (ICP) cells to determine the differential delay between the links in the IMA group. ICP cells are sent on each link once per IMA frame. By default, each frame consists of 128 cells. The ICP cell can be located anywhere inside the frame (cell position 0 or cell position 127 or any cell position in between).
Each frame includes an IMA frame sequence number in the ICP cell. This number increments per link and with each transmitted IMA frame. Over the two or more links in the IMA group, the sequence number in the ICP cell of each link is identical. In other words, link 0 uses IMA sequence 0 at the same time that link 1 uses IMA sequence number 0.
The IMA technical specifications on the ATM Forum define how both the transmitter and the receiver act with respect to differential link delay among the constituent links in the IMA group.
-
Transmitter’s requirements—The transmit IMA shall not introduce more than 2.5 cell times at the physical link rate of differential delay among the constituent links.
-
Receiver’s requirements—The amount of link differential delay tolerated by an IMA implementation shall be up to at least 25 milliseconds when used over DS-1 or E1 links. The amount of link differential delay tolerance may be configured up to the maximum value supported by the IMA implementation.
Both ends of the IMA virtual link may be configured with different amounts of tolerable differential delay.
Appendix A of the standard provides an example of how the receiving IMA virtual link compensates for differential delay. At the receive end, each physical link has its own circular buffer which should be deep enough to tolerate the maximum differential delay variation.
The next diagram shows the cells of three physical links in the same IMA group. The transmitting IMA process distributes ATM cells over the links in a cyclic, round-robin fashion and on a cell-by-cell basis. Whenever a cell is written into the buffer, the write pointer increments. A read (playback) pointer increments as cells are read out of the buffer. All of the delay compensation is done by adjusting the write pointer. IMA delay compensation changes the virtual alignment of the user cells so that, with each read interval or IMA clock tick, one data cell is removed from the buffer. The receiving IMA process always reads one cell per link in a cyclic, round-robin fashion. One cell time equals 276 microseconds at the DS-1 rate. Thus, assuming a maximum allowable delay of 25 milliseconds, the maximum difference acceptable between the write pointers is 91 cells.
In this sample transmission, the source interface has staggered the ICP cells in various cell positions within an IMA frame. At the destination interface, Link 0 and Link 2 have the same amount of propagation delay, but Link 1 has a delay one cell time longer than Link 0 or Link 2.
IMA MIB Values for Differential Delay
The IMA recommendation defines a MIB for collecting IMA-specific statistics for the member T1s and for the IMA group itself. The IMA MIB provides these counters, which are relevant in troubleshooting problems related to differential delay:
-
ImaGroupLeastDelayLink—Interface index (ifIndex) of the link configured in the IMA group which has the smallest link propagation delay. The distinguished value of zero may be used if no link has been configured in the IMA group, or if the link with the smallest link propagation delay has not yet been determined.
-
ImaGroupDiffDelayMaxObs—Latest maximum differential delay observed (in milliseconds) between the links with the least and the most link propagation delay, among the receive links that are currently configured in the IMA group.
Viewing Configured and Measured Values on NM-IMA
On the IMA network module for Cisco 2600 and 3600 series routers, issue the show ima interface atm command to view the configured maximum allowed differential delay value for your IMA group.
2600# show ima interface atm2/ima2 Interface atm2/IMA2 is up Group index is 2 Ne state is operational, failure status is noFailure active links bitmap 0x30 IMA Group Current Configuration: TX/Rx configured links bitmap 0x30/0x30 TX/Rx minimum required links 1/1 Maximum allowed diff delay is 25ms, TX frame length 128 NE TX clock mode CTC, configured timing reference link atm2/4 Test pattern procedure is disabled IMA Group Current Counters (time elapsed 12 seconds): 3 NE Failures, 3 Fe Failures, 4 Unavail Secs IMA Group Total Counters (last 0 15 minute intervals): 0 NE Failures, 0 Fe Failures, 0 Unavail Secs IMA link Information: Physical Status NearEnd Rx Status Test Status ------------ --------------- ----------------- ----------- atm2/4 up active disabled atm2/5 up active disabled
Issue the show ima interface atm command on the physical T1 member interface to view the currently measured value. This command also displays IMA link error counters for the physical interface. Refer to Troubleshooting ATM IMA Links on Cisco 2600 and 3600 Routers for an explanation of these error counters.
3640-2.2# show ima interface atm 0/0 Interface ATM0/0 is up ifIndex 1, Group Index 1, Row Status is active TX/Rx Lid 0/0, relative delay 0ms NE TX/Rx state active/active Fe TX/Rx state active/active NE Rx failure status is noFailure Fe Rx failure status is noFailure Rx test pattern 0x40, test procedure disabled IMA Link Current Counters (time elapsed 866 seconds): 0 Ima Violations, 0 Oif Anomalies 0 NE Severely Err Secs, 0 Fe Severely Err Secs 0 NE Unavail Secs, 0 Fe Unavail Secs 0 NE TX Unusable Secs, 0 NE Rx Unusable Secs 0 Fe TX Unusable Secs, 0 Fe Rx Unusable Secs 0 NE TX Failures, 0 NE Rx Failures 0 Fe TX Failures, 0 Fe Rx Failures !--- Output suppressed.
Finally, issue the show ima interface atm slot/imagroup-number detail command to view the maximum observed differential delay value among the member interfaces and to view the particular member interface that is exhibiting the least amount of delay.
3640-2.2# show ima interface atm0/ima0 detail Interface ATM0/IMA0 is up Group index is 1 NE state is operational, failure status is noFailure Active links bitmap 0x3 IMA Group Current Configuration: TX/Rx configured links bitmap 0x3/0x3 TX/Rx minimum required links 1/1 Maximum allowed diff delay is 25ms, TX frame length 128 NE TX clock mode CTC, configured timing reference link ATM0/0 Test pattern procedure is disabled Detailed group Information: TX/Rx Ima_id 0x0/0x0, symmetry symmetricOperation Number of TX/Rx configured links 2/2 Number of TX/Rx active links 2/2 Fe TX clock mode ctc, Rx frame length 128 TX/Rx timing reference link 0/0 Maximum observed diff delay 0ms, least delayed link 0 Running seconds 101257 GTSM last changed 23:35:52 UTC Sat Mar 6 1993 IMA Group Current Counters (time elapsed 896 seconds): 0 NE Failures, 0 Fe Failures, 0 Unavail Secs IMA Group Interval(1) Counters: 0 NE Failures, 0 Fe Failures, 0 Unavail Secs IMA Group Interval(2) Counters: 0 NE Failures, 0 Fe Failures, 0 Unavail Secs
Viewing Configured and Measured Values on PA-A3-IMA
On a PA-A3-8T1 or E1 IMA port adapter, issue the show controllers atm command to see the configured value.
atm# show controllers atm1/ima0 Interface ATM1/IMA0 is down Hardware is ATM IMA !--- Output suppressed. ATM channel number is 2 link members are 0x4, active links are 0x0 Group status is blockedNe, 1 links configured, Group Info: Configured links bitmap 0x4, Active links bitmap 0x0, TX/Rx IMA_id 0x11/0x63, NE Group status is startUp, frame length 0x80, Max Diff Delay 50, 1 min links, clock mode ctc, symmetry symmetricOperation, trl 2, Group Failure status is startUpNe. Test pattern procedure is disabled
On a PA-A3-8T1 or E1 IMA port adapter, issue the show ima interface atm slot/imagroup-number detail command to view the currently measured values for differential delay.
7200# show ima interface atm 1/ima0 detail ATM1/ima0 is up ImaGroupState:NearEnd = operational, FarEnd = operational ImaGroupFailureStatus = noFailure IMA Group Current Configuration: ImaGroupMinNumTxLinks = 2 ImaGroupMinNumRxLinks = 2 ImaGroupDiffDelayMax = 25 ImaGroupNeTxClkMode = common(ctc) ImaGroupFrameLength = 128 ImaTestProcStatus = disabled ImaGroupTestLink = 0 ImaGroupTestPattern = 0xFF IMA MIB Information: ImaGroupSymmetry = symmetricOperation ImaGroupFeTxClkMode = common(ctc) ImaGroupRxFrameLength = 128 ImaGroupTxTimingRefLink = 0 ImaGroupRxTimingRefLink = 0 ImaGroupTxImaId = 0 ImaGroupRxImaId = 0 ImaGroupNumTxCfgLinks = 2 ImaGroupNumRxCfgLinks = 2 ImaGroupNumTxActLinks = 2 ImaGroupNumRxActLinks = 2 ImaGroupLeastDelayLink = 1 ImaGroupDiffDelayMaxObs = 0 IMA group counters: ImaGroupNeNumFailures = 78 ImaGroupFeNumFailures = 68 ImaGroupUnAvailSecs = 441453 ImaGroupRunningSecs = 445036
IMA Group Comes Down
Every IMA group must have a minimum number of links to be active. Issue the ima active-links-minimum number command to change this value. If the number of active links falls below the configured value, your router legitimately brings down the IMA virtual interface.
If your router’s IMA group interface goes down unexpectedly, determine what equipment terminates the IMA protocol at the remote end of the link before contacting Cisco Technical Support. Cisco Bug ID CSCdr93036 (duplicated to CSCdr19633) documents a problem with the Processor Switching Module (PXM) and the AUSMB-8T1, which provides IMA services for the Cisco MGX WAN switch. In rare conditions, one physical T1 link coming down brings down the entire IMA group.
If the IMA virtual interface is assigned a number as a group name and a physical link with the same number goes down, the IMA group comes down. In this sample output, the AUSMB-8 card is configured with an IMA group identified with the value of 1. The group consists of six links identified by link numbers 1 through 6.
MGXISH03.1.29.AUSMB8.a > dspimainfo
Link Group NeTx NeRx FeTx FeRx TxLID RxID
State State State State
-------------------------------------------------
1 1 Active Active Active Active 0 0
2 1 Active Active Active Active 1 1
3 1 Active Active Active Active 2 2
4 1 Active Active Active Active 3 3
5 1 Active Active Active Active 4 4
6 1 Active Active Active Active 5 5
The root cause of the problem was that the line number going to alarm generated an interface trap (note that the other lines going into alarm did not generate such a trap).
An ATM user service module (AUSM) generates a port trap to the PXM when a line goes down and there is a port within the line. The fix changes the behavior of the AUSM and ensures that the AUSM sends a port trap only when the number of active links falls below the required minimum.
Known Issues
In rare circumstances, an IMA group may experience the removal of multiple T1 physical links, as reported in the router’s log.
!--- Each of these timestamped lines of output appear on one line. APR 2 13:57:17: %IMA-5-ACTIVE_LINK_CHANGE: IMA Group ATM1/IMA1 now has 4 active links, active link bitmap is 0xD. APR 2 13:57:17: %IMA-5-ACTIVE_LINK_CHANGE: IMA Group ATM1/IMA1 now has 1 active links, active link bitmap is 0xD. APR 2 13:57:17: %IMA-5-ACTIVE_LINK_CHANGE: IMA Group ATM1/IMA1 now has 4 active links, active link bitmap is 0xD. APR 2 13:57:18: %IMA-5-ACTIVE_LINK_CHANGE: IMA Group ATM1/IMA1 now has 3 active links, active link bitmap is 0xD.
These symptoms match Cisco Bug ID CSCdr39332, which documents a rare condition in which exceeding the configured maximum differential delay value causes the IMA process to remove the good links or to bring down the entire IMA group. The expected action is that the delay link comes down. The problem is related to how the IMA firmware uses the delay compensation buffer and adjusts the write pointers, as described in this document. As a workaround, try increasing the configured maximum value with the ima differential-delay-maximum milliseconds command. Specify a value from 25 to 250 milliseconds.
If you experience this issue, collect output from the show log and show tech commands and open a service request with Cisco Technical Support.
Related Information
Revision History
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
06-Sep-2018 |
Initial Release |
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)