General Troubleshooting

This chapter provides procedures for troubleshooting the most common problems encountered when operating the NCS 1020 chassis.

Troubleshooting Client-Side Failure

Problem

This problem occurs when there is a signal payload loss in the following conditions:

  • If the fiber cut occurs at the client-side in the direction from the client to the NCS 1020, the LOS-P alarm is raised at the AD port controller OTS.

  • If the fiber cut occurs at the client-side in the direction from the NCS 1020 to the client, the LOS-P alarm is raised at the Passive device controller OMS.

Topology and Components

To build this topology, you need the following hardware:

  • Cisco NCS 1020 devices

  • QDD-400G-ZR-S transceiver

The following figure displays client-side failure due to fiber cut:

Figure 1. Fiber Cut on Line

Example 1

The following example shows how to determine LOS-P failure at the AD port controller OTS:

RP/0/RP0/CPU0:ios#show controllers ots0/0/0/3
Tue Mar 28 15:58:12.210 UTC

Controller State: Down 

Transport Admin State: Automatic In Service 

LED State: Red 

         Alarm Status:
         -------------
         Detected Alarms: 
                 RX-LOS-P    

         Alarm Statistics:
         -----------------
         RX-LOS-P = 2          
         RX-LOC = 0          
         TX-POWER-FAIL-LOW = 0          
         INGRESS-AUTO-LASER-SHUT = 0          
         INGRESS-AUTO-POW-RED = 0          
         INGRESS-AMPLI-GAIN-LOW = 0          
         INGRESS-AMPLI-GAIN-HIGH = 0          
         EGRESS-AUTO-LASER-SHUT = 0          
         EGRESS-AUTO-POW-RED = 0          
         EGRESS-AMPLI-GAIN-LOW = 0          
         EGRESS-AMPLI-GAIN-HIGH = 0          
         HIGH-TX-BR-PWR = 0          
         HIGH-RX-BR-PWR = 0          
         SPAN-TOO-SHORT-TX = 0          
         SPAN-TOO-SHORT-RX = 0          

         Parameter Statistics:
         ---------------------
         Total Rx Power = -50.00 dBm 
         Total Tx Power = -50.00 dBm 
         Ingress Ampli Mode = Gain
         Ingress Ampli Gain = 0.0 dB
         Ingress Ampli Tilt = 0.0 dB



         Configured Parameters:
         -------------
         Ingress Ampli Mode = Gain
         Ingress Ampli Gain = 16.0 dB 
         Ingress Ampli Power = -3.0 dBm
         Ingress Ampli Tilt = 0.0 dB
         Rx Low Threshold = -3276.8 dBm 
         Tx Low Threshold = -3276.8 dBm

Example 2

The following example shows how to determine LOS-P failure at the passive device controller OMS:

RP/0/RP0/CPU0:P2A_DT_07#sh controllers oms 0/1/0/8
Tue Mar 28 16:00:33.685 UTC

Controller State: Down 

Transport Admin State: Automatic In Service 

LED State: Red 

         Alarm Status:
         -------------
         Detected Alarms: 
                 RX-LOS-P    

         Alarm Statistics:
         -----------------
         RX-LOS-P = 1          
         TX-POWER-FAIL-LOW = 0          

         Parameter Statistics:
         ---------------------
         Tx Power = -50.00 dBm 
         Rx Power = -50.00 dBm

Solution

See RX-LOS-P to resolve this alarm.

Troubleshooting Line Port SPAN Failure

Problem

This problem occurs when there is a loss of fiber connection at the Line port due to fiber cut. As a result, the LOC alarm is raised at the Line OTS controller.

Topology and Components

To build this point-to-point topology, you need the following hardware:

  • Cisco NCS 1020 devices

  • NCS1K-MD32E-C modules

  • NCS1K-MD32O-C modules

  • QDD-400G-ZR-S transceiver

The following figure shows loss of fiber connection continuity at the Line port:

Figure 2. Fiber Cut at the Line OTS Controller

Example 1

The following example shows how to determine a span failure.

RP/0/RP0/CPU0:ios#show controllers ots0/0/0/0
Tue Mar 28 15:48:31.486 UTC

Controller State: Down 

Transport Admin State: In Service 

LED State: Red 

         Alarm Status:
         -------------
         Detected Alarms: 
                 RX-LOS-P    
                 RX-LOC    
                 EGRESS-AUTO-LASER-SHUT    

         Alarm Statistics:
         -----------------
         RX-LOS-P = 2          
         RX-LOC = 2          
         TX-POWER-FAIL-LOW = 3          
         INGRESS-AUTO-LASER-SHUT = 0          
         INGRESS-AUTO-POW-RED = 0          
         INGRESS-AMPLI-GAIN-LOW = 0          
         INGRESS-AMPLI-GAIN-HIGH = 0          
         EGRESS-AUTO-LASER-SHUT = 2          
         EGRESS-AUTO-POW-RED = 0          
         EGRESS-AMPLI-GAIN-LOW = 0          
         EGRESS-AMPLI-GAIN-HIGH = 0          
         HIGH-TX-BR-PWR = 0          
         HIGH-RX-BR-PWR = 0          
         SPAN-TOO-SHORT-TX = 0          
         SPAN-TOO-SHORT-RX = 0          

         Parameter Statistics:
         ---------------------
         Total Rx Power(C+L) = 10.69 dBm
         Total Tx Power(C+L) = 19.50 dBm
         Total Rx Power = -50.00 dBm 
         Total Tx Power = -50.00 dBm 
         Rx Signal Power = -50.00 dBm 
         Tx Signal Power = -50.00 dBm 
         BR Power = -90.00 dBm 
         Tx Voa Attenuation = 15.0 dB 
         Ingress Ampli Mode = Gain
         Ingress Ampli Gain = 0.0 dB
         Ingress Ampli Tilt = 0.0 dB
         Ingress Ampli Gain Range = Normal 
         Ingress Ampli Safety Control mode = auto 
         Ingress Ampli OSRI = OFF 
         Ingress Ampli Force Apr = OFF 
         Egress Ampli Mode = Gain
         Egress Ampli Gain = 0.0 dB
         Egress Ampli Tilt = 0.0 dB
         Egress Ampli Safety Control mode = auto 
         Egress Ampli OSRI = OFF 
         Egress Ampli Force APR = OFF 
         Egress Ampli BR = ENABLE 

         Configured Parameters:
         -------------
         Tx Voa Attenuation = 15.0 dB 
         Ingress Ampli Mode = Gain
         Ingress Ampli Gain = 12.0 dB 
         Ingress Ampli Power = -1.0 dBm
         Ingress Ampli Tilt = 0.0 dB
         Ingress Ampli Gain Range = Normal 
         Ingress Ampli Safety Control mode = auto 
         Ingress Ampli OSRI = OFF 
         Ingress Ampli Force Apr = OFF 
         Egress Ampli Mode = Gain
         Egress Ampli Gain = 16.0 dB 
         Egress Ampli Power = 5.0 dBm
         Egress Ampli Tilt = 0.0 dB
         Egress Ampli Safety Control mode = auto 
         Egress Ampli OSRI = OFF 
         Egress Ampli Force APR = OFF 
         Egress Ampli BR = ENABLE 
         BR High Threshold = -17.0 dBm 
         Rx Low Threshold = -3276.8 dBm 
         Tx Low Threshold = -3276.8 dBm

Solution

See RX-LOC Alarm to resolve this problem.

Troubleshooting Power Oscillation in the Network

Problem

This problem occurs when there is a power oscillation in the network.

Example 1

The following example shows the power fluctuation in the network:

RP/0/RP0/CPU0:ios#show controllers ots0/0/0/0 pm current 30-sec optics 1 
Tue Mar 28 16:02:22.332 UTC

Optics in the current interval [16:02:00 - 16:02:22 Tue Mar 28 2023]

Optics current bucket type : Valid
              MIN        AVG      MAX      Operational    Configured      TCA   Operational     Configured     TCA
                                          Threshold(min)  Threshold(min) (min) Threshold(max)   Threshold(max) (max)
OPT[dBm]      : -50.00   -7.32    6.10     -20.00            NA              NO   40.00            NA              NO   
OPR[dBm]      : -50.00   -50.00   -50.00   -30.00            NA              NO   18.00            NA              NO   
OPT(C+L)[dBm] : 19.50    19.51    19.70    -20.00            NA              NO   60.85            NA              NO   
OPR(C+L)[dBm] : 10.60    10.64    10.70    -30.00            NA              NO   36.00            NA              NO   
OPT(S)[dBm]   : -50.00   -50.00   -50.00   -5.00             NA              NO   28.00            NA              NO   
OPR(S)[dBm]   : -50.00   -50.00   -50.00   -30.00            NA              NO   18.00            NA              NO  
 OPBR[dBm]    : -50.00   -50.00   -50.00   -30.00            NA              NO   -14.09           NA              NO   
EAGN[dB]      : 0.00     0.00     0.00     16.00             NA              NO   31.00            NA              NO   
EATL[dB]      : 0.00     0.00     0.00     -5.00             NA              NO   5.00             NA              NO   
IAGN[dB]      : 0.00     0.00     0.00     12.00             NA              NO   25.00            NA              NO   
IATL[dB]      : 0.00     0.00     0.00     -5.00             NA              NO   5.00             NA              NO   

Last clearing of "show controllers OPTICS" counters never

Solution

You need to monitor the performance monitor statistics of the corresponding OTS and OTS-OCH controllers to determine where the power oscillation occurs in the network. Check for the transient alarms on the node, such as RX-LOC and RX-LOS-P. See RX-LOS-P and RX-LOC Alarm to resolve this problem.

Troubleshooting Connection Verification Failures

sysdb_item_set_failed

The sysdb_item_set_failed error message is a warning condition that the system reports when incorrect configurations are made during connection verification. This message appears for the following conditions:

  1. Tone Generation Already In Progress

  2. Tone Generation Has Invalid Pattern

  3. Tone Generation Has Invalid Frequency

  4. Tone Generation Start Failed

  5. Tone Detection Has Invalid Pattern-Expected

Error Message 1:

sysdb_item_set_failed('optics' detected the 'warning' condition 'Tone Generation Already In Progress')

Tone Generation Already In Progress appears when an attempt was made to start tone generation on one controller without stopping the tone generation operation on another controller.

The following example shows that the tone generation is initiated on port 5 OTS controller (ots0/0/0/5) while tone generation on port 4 OTS controller (ots0/0/0/4) is still in progress.

Example
RP/0/RP0/CPU0:ios#tone-pattern controller ots 0/0/0/4 start 
Fri Sep 15 14:26:18.670 UTC
Tone pattern started
RP/0/RP0/CPU0:ios#tone-pattern controller ots 0/0/0/5 start 
Fri Sep 15 14:27:20.418 UTC
sysdb_item_set_failed('optics' detected the 'warning' condition 'Tone Generation Already In Progress')

Solution

If you want to continue tone generation in the new OTS controller, then follow the following procedure to stop the tone generation that is already in progress.

  1. Find the OTS controller that currently has tone generation in progress. To find, run the show alarms brief system active or show alarms brief system conditions command.

    The following example shows that tone generation is active on the port 4 OTS controller.

    Example
    RP/0/RP0/CPU0:ios#show alarms brief system active
    Thu Nov 23 06:42:20.077 UTC
    
    ------------------------------------------------------------------------------------
    Active Alarms 
    ------------------------------------------------------------------------------------
    Location        Severity     Group            Set Time                   Description
    ------------------------------------------------------------------------------------
    0/PM1           Major        Environ          11/17/2023 04:11:37 UTC    Power Module Error (PM_VIN_VOLT_OOR)  
    0/PM1           Major        Environ          11/17/2023 04:11:37 UTC    Power Module Output Disabled (PM_OUTPUT_DISABLED)
    0               Major        Environ          11/17/2023 04:11:37 UTC    Power Group redundancy lost                      
    0/RP0/CPU0      Minor        Software         11/17/2023 04:12:39 UTC    SW Upgrade is still allowed as SIA Grace Period is remaining 
    0/0             Minor        Controller       11/23/2023 02:34:38 UTC    Ots0/0/0/4 - Tone Generation In Progress
    0/0             Critical     Controller       11/23/2023 06:41:07 UTC    Ots0/0/0/14 - Loss of Signal - Payload 
  2. Stop the tone generation in that OTS controller. To stop, run the tone-pattern controller ots 0/0/0/4 stop command.

    The following example shows that tone pattern generation is stopped on the port 4 OTS controller.

    Example
    RP/0/RP0/CPU0:ios#tone-pattern controller ots 0/0/0/4 stop
    Tue May 10 11:50:45.837 UTC
    Tone pattern stopped

Error Message 2:

sysdb_item_set_failed('optics' detected the 'warning' condition 'Tone Generation Has Invalid Pattern')

The Tone Generation Has Invalid Pattern error message appears when an attempt was made to start the tone generation on the OTS controller without configuring the tone-pattern value for the controller.

The following example does not display tone-pattern for the show run controller ots 0/0/0/4 command. Due to the absence of tone-pattern, the system reports the Tone Generation Has Invalid Pattern message.

Example
RP/0/RP0/CPU0:ios#show run controller ots 0/0/0/33
Fri Nov  3 10:17:01.999 UTC
controller Ots0/0/0/33
 tone-rate 25
 tone-frequency 196
!
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        RP/0/RP0/CPU0:Enceladus_1#tone-pattern controller ots 0/0/0/33 start 
Fri Nov  3 10:17:46.389 UTC
sysdb_item_set_failed('optics' detected the 'warning' condition 'Tone Generation Has Invalid Pattern')
RP/0/RP0/CPU0:ios

Solution

Set the tone-pattern value for the OTS controller. To set, run the following command:

Command Example
RP/0/RP0/CPU0:(config-Ots)#tone-pattern abcd1234
RP/0/RP0/CPU0:(config-Ots)#commit

Error Message 3:

sysdb_item_set_failed('optics' detected the 'warning' condition 'Tone Generation Has Invalid Frequency')

The Tone Generation Has Invalid Frequency error message appears when an attempt was made to start tone generation without configuring the tone-frequency value in the OTS controller.

The following example does not display tone-frequency for the show run controller ots 0/0/0/4 command. Due to the absence of tone-frequency, the system reports the Tone Generation Has Invalid Frequency message.

Example
RP/0/RP0/CPU0:ios#show run controller ots 0/0/0/4
Fri Sep 15 14:37:03.381 UTC
controller Ots0/0/0/4
 tone-rate 2
 tone-pattern abcd1234
 tone-detect-oob
 tone-pattern-expected abcd1234
!

RP/0/RP0/CPU0:ios#tone-pattern controller ots 0/0/0/4 start
Fri Sep 15 14:37:41.777 UTC
sysdb_item_set_failed('optics' detected the 'warning' condition 'Tone Generation Has Invalid Frequency')
RP/0/RP0/CPU0:ios#

Solution

Set the tone-frequency value in out-of-band (OOB) frequency for the OTS controller. To set, run the following command:

Command Example
RP/0/RP0/CPU0:(config-Ots)#tone-frequency 191.175
RP/0/RP0/CPU0:(config-Ots)#commit

Error Message 4:

sysdb_item_set_failed('optics' detected the 'warning' condition 'Tone Generation Start Failed')

The Tone Generation Start Failed error message appears when an attempt was made to start tone generation using an in-band frequency of a provisioned channel.

The following example shows that tone-frequency is configured for the show run controller ots 0/0/0/33 command. However, the tone-frequency value that is configured is the central frequency (191.375) of the provisioned channel 1. Hence, the system reports the Tone Generation Start Failed message.

Example
RP/0/RP0/CPU0:ios#show hw-module location 0/0/NXR0 terminal-ampli 
Fri Nov  3 10:31:25.329 UTC

Legend:
NXC      - Channel not cross-connected
ACTIVE - Channel cross-connected to data port
ASE        - Channel filled with ASE
FAILED - Data channel failed, pending transition to ASE

Location:             0/0/NXR0

Status:               Provisioned

Flex Grid Info

Channel Number      Centre Frequency(THz)       Channel Width(GHz)    Channel Status    Overlapping Channels
1                   191.375000                  75.000                ACTIVE              - ,   2
2                   191.435000                  75.000                ASE                 1 ,   -

RP/0/RP0/CPU0:ios#show run controller ots 0/0/0/33
Fri Nov  3 10:38:44.414 UTC
controller Ots0/0/0/33
 tone-rate 25
 tone-pattern abcd1234
 tone-frequency 191.375
!

RP/0/RP0/CPU0:ios#tone-pattern controller ots 0/0/0/33 start 
Fri Nov  3 10:39:03.532 UTC
sysdb_item_set_failed('optics' detected the 'warning' condition 'Tone Generation Start Failed')
RP/0/RP0/CPU0:ios#

Solution

Set the tone-frequency value in the out-of-band (OOB) frequency for the OTS controller. To set, run the following command:

Command Example
RP/0/RP0/CPU0:(config-Ots)#tone-frequency 191.175
RP/0/RP0/CPU0:(config-Ots)#commit

Error Message 5:

sysdb_item_set_failed('optics' detected the 'warning' condition 'Tone Detection Has Invalid Pattern-Expected')

The Tone Detection Has Invalid Pattern-Expected error message appears when an attempt was made to start tone generation without configuring the expected tone pattern in the OTS, OMS, or OCH controller.

The following example shows the show run controller ots 0/0/0/33 output without the tone-pattern-expected value. Due to the absence of tone-pattern-expected, the system reports the Tone Detection Has Invalid Pattern-Expected message.

Example
RP/0/RP0/CPU0:ios#show run controller ots 0/0/0/33
Fri Nov  3 10:45:29.171 UTC
controller Ots0/0/0/33
 tone-rate 25
 tone-pattern abcd1234
 tone-frequency 191.375
!

RP/0/RP0/CPU0:ios#tone-pattern-detect controller ots 0/0/0/33 start 
Fri Nov  3 10:46:01.585 UTC
sysdb_item_set_failed('optics' detected the 'warning' condition 'Tone Detection Has Invalid Pattern-Expected')
RP/0/RP0/CPU0:ios#

Solution

Set the tone-pattern-expected value on the controller that is expecting the tone pattern. In this example, the tone pattern is expected on the OTS controller. To set, run the following command:

Command Example
RP/0/RP0/CPU0:(config-Ots)#tone-pattern-expected abcd1234
RP/0/RP0/CPU0:(config-Ots)#commit

Failed to Commit One or More Configuration Items During a Pseudo-Atomic Operation.

The NCS 1020 CLI fails to commit connection verification configuration when an attempt was made to set an incorrect value or parameter on an OTS controller.

In the following examples, the Failed to commit one or more configuration... error message appears to indicate that controller configuration has failed.

Example 1:

Example
RP/0/RP0/CPU0:ios#configure
Fri Sep 15 14:38:18.254 UTC
RP/0/RP0/CPU0:ios(config)#controller ots 0/0/0/4
RP/0/RP0/CPU0:ios(config-Ots)#tone-frequency 30
RP/0/RP0/CPU0:ios(config-Ots)#commit 
Fri Sep 15 14:38:50.918 UTC

% Failed to commit one or more configuration items during a pseudo-atomic operation. All changes made have been reverted. Please issue 'show configuration failed [inheritance]' from this session to view the errors

Solution 1:

Run the show configuration failed command to find the cause for the system to fail the OTS controller configuration.

Command
RP/0/RP0/CPU0:ios(config-Ots)#show configuration failed 
Fri Sep 15 14:38:57.773 UTC
!! SEMANTIC ERRORS: This configuration was rejected by 
!! the system due to semantic errors. The individual 
!! errors with each failed configuration command can be 
!! found below.


controller Ots0/0/0/4
 tone-frequency 30
!!% Invalid argument: Entered value not supported. Range supported by the platform is from 191.175000 to 196.100000
!
end

The preceding output shows that an unsupported value is entered for tone-frequency. The output also suggests the frequency range (191.175000…196.100000) that the platform supports for connection verification.

Example 2:

Example
RP/0/RP0/CPU0:ios#configure
Fri Sep 15 14:45:44.275 UTC
RP/0/RP0/CPU0:ios(config)#controller ots 0/0/0/4
RP/0/RP0/CPU0:ios(config-Ots)#tone-pattern ?
  LINE  Enter Hex Values(Max Length 64 Bytes)
RP/0/RP0/CPU0:ios(config-Ots)#tone-pattern xxxyyyzzz
RP/0/RP0/CPU0:ios(config-Ots)#commit 
Fri Sep 15 14:46:20.073 UTC

% Failed to commit one or more configuration items during a pseudo-atomic operation. All changes made have been reverted. Please issue 'show configuration failed [inheritance]' from this session to view the errors

Solution 2:

Command
RP/0/RP0/CPU0:iosconfig-Ots)#show configuration failed 
Fri Sep 15 14:46:39.396 UTC
!! SEMANTIC ERRORS: This configuration was rejected by 
!! the system due to semantic errors. The individual 
!! errors with each failed configuration command can be 
!! found below.


controller Ots0/0/0/4
 tone-pattern xxxyyyzzz
!!% Invalid argument: invalid hex value,valid value is 0-9 a-f A-F
!
end

The preceding output shows that an unsupported value is entered for tone-pattern. The output also suggests the supported values for tone pattern.

Example 3:

Example
RP/0/RP0/CPU0:ios#configure
Fri Sep 15 15:14:36.131 UTC
RP/0/RP0/CPU0:ios(config)#controller och 0/3/0/1
RP/0/RP0/CPU0:ios(config-Och)#tone-detect-oob 
RP/0/RP0/CPU0:ios(config-Och)#commit 
Fri Sep 15 15:14:57.286 UTC

% Failed to commit one or more configuration items during a pseudo-atomic operation. All changes made have been reverted. Please issue 'show configuration failed [inheritance]' from this session to view the errors

Solution 3:

Command
RP/0/RP0/CPU0:ios(config-Och)#show configuration failed 
Fri Sep 15 15:15:06.219 UTC
!! SEMANTIC ERRORS: This configuration was rejected by 
!! the system due to semantic errors. The individual 
!! errors with each failed configuration command can be 
!! found below.


controller Och0/3/0/1
 tone-detect-oob
!!% Invalid argument: Tone Detect OOB is not configurable for this controller 
!
end

The preceding output informs that Tone Detect OOB is not configurable on OCH controllers. Configure OOB only on OMS controllers.

Invalid Input Detected at '^' Marker

The Invalid input detected at '^' marker error message appears when an attempt was made to start tone generation on the OMS and OCH controllers. Tone generation must be initiated on the OTS controller. The OMS and OCH controllers should be used only to detect the tone that is generated from the OTS controller.

The following example shows the system reporting the Invalid input detected at '^' marker error message on the port 0 OMS controller (oms 0/3/0/0).

Example
RP/0/RP0/CPU0:ios#tone-pattern controller oms 0/3/0/0 start
                                                       ^
% Invalid input detected at '^' marker.
RP/0/RP0/CPU0:ios#

The following example shows the system reporting the Invalid input detected at '^' marker error message on the port 1 OCH controller (och 0/3/0/1).

Example
RP/0/RP0/CPU0:ios#tone-pattern controller och 0/3/0/1 start
                                                       ^
% Invalid input detected at '^' marker.

Solution

Use the OMS and OCH controllers for detecting the tone generation only. See for OCH and OMS controller syntax and parameters.