Inter-rack Timing

This chapter provides the details about inter-rack timing in the Cisco NCS 4000 Series Router.

Introduction

In a MC system the source and destination ports of the cross connect can be across racks. Inter-rack (or cross-rack) timing allows the timing information to be passed across racks for segmentation and re-assembly needs.

Verification of Inter-rack Timing

Procedure

Step 1

Verify the inter-rack timing configuration, using command show running-config frequency synchronization

Example:


RP/2/RP0:MC_FLT+4+1# show running-config frequency synchronization 
Thu Mar 22 11:33:30.986 IST
frequency synchronization
clock-interface timing-mode system
Step 2

Verify FPD Status for Timing-FPGA and ECU-FPGA, using command show hw-module fpd <fpd-name>

Example:


RP/2/RP0:MC_FLT+4+1#show hw-module fpd Timing-FPGA
Thu Mar 22 13:47:18.695 IST
                        FPD Versions
                      =================
Location   Card type        HWver FPD device       ATR Status   Running Programd
------------------------------------------------------------------------------
0/RP0     NCS4K-RP          0.1   Timing-FPGA       S  CURRENT    3.82    3.82  
0/RP1     NCS4K-RP          0.1   Timing-FPGA       S  CURRENT    3.82    3.82  
1/RP0     NCS4K-RP          0.1   Timing-FPGA       S  CURRENT    3.82    3.82  
1/RP1     NCS4K-RP          0.1   Timing-FPGA       S  CURRENT    3.82    3.82  
2/RP0     NCS4K-RP          0.1   Timing-FPGA       S  CURRENT    3.82    3.82  
2/RP1     NCS4K-RP          0.1   Timing-FPGA       S  CURRENT    3.82    3.82  
3/RP0     NCS4K-RP          0.1   Timing-FPGA       S  CURRENT    3.82    3.82  
3/RP1     NCS4K-RP          0.1   Timing-FPGA       S  CURRENT    3.82    3.82  
 


RP/2/RP0:MC_FLT+4+1# show hw-module fpd ECU-FPGA   
Thu Mar 22 13:47:25.868 IST
                       FPD Versions
                      =================
Location   Card type        HWver FPD device       ATR Status   Running Programd
------------------------------------------------------------------------------
0/EC0     NCS4K-ECU2        0.2   ECU-FPGA             CURRENT    4.08    4.08  
1/EC0     NCS4K-ECU2        0.2   ECU-FPGA             CURRENT    4.08    4.08  
2/EC0     NCS4K-ECU2        0.2   ECU-FPGA             CURRENT    4.08    4.08  
3/EC0     NCS4K-ECU2        0.2   ECU-FPGA             CURRENT    4.08    4.08  
RP/2/RP0:MC_FLT+4+1#
Step 3

Verify that all the FPDs on LC are in CURRENT state, using command show hw-module location <LC location> fpd

Example:

RP/2/RP0:MC_FLT+4+1# show hw-module location 0/5 fpd
                       FPD Versions
                     =================
Location   Card type        HWver FPD device       ATR Status   Running Programd
------------------------------------------------------------------------------
0/5       NCS4K-2H10T-OP-KS 0.2   Backup-ZYNQ      BSP CURRENT                  
0/5       NCS4K-2H10T-OP-KS 0.2   CCC-FPGA             CURRENT    1.50    1.50  
0/5       NCS4K-2H10T-OP-KS 0.2   CCC-Power-On         CURRENT    1.14    1.14  
0/5       NCS4K-2H10T-OP-KS 0.2   DIGI1                CURRENT    2.03    2.03  
0/5       NCS4K-2H10T-OP-KS 0.2   DIGI2                CURRENT    2.03    2.03  
0/5       NCS4K-2H10T-OP-KS 0.2   Ethernet-Switch      CURRENT    1.02    1.02  
0/5       NCS4K-2H10T-OP-KS 0.2   GRIMA                CURRENT    1.51    1.51  
0/5       NCS4K-2H10T-OP-KS 0.2   PLX-8649             CURRENT    0.11    0.11
Step 4

Verify Slice Manager Status for all Active LC VM's, using command show controllers slice-control all location <location>

Note 

Additionally verify that the Clock Status on all LCs is External.

Example:


RP/2/RP0:MC_FLT+4+1# show controllers slice-control all location 0/LC1
Thu Mar 22 14:36:42.685 IST
CARD 0 IS OFFLINE
CARD 1 IS OFFLINE
CARD 3 IS OFFLINE
CARD 8 IS OFFLINE
CARD 10 IS OFFLINE
CARD 11 IS OFFLINE
CARD 12 IS OFFLINE
CARD 13 IS OFFLINE
CARD 14 IS OFFLINE
==============================================
Slice Controller Context: 2
==============================================
Inserted              : Yes
Physical Slot number  : 3
Logical slot number   : 2
Board type            : 5408a5 (BOARD_TYPE_SCAPA_1x100GE_CPAK_10x10GE)
Slice oper state      : OPERATIONAL
Bao Version           : 0.1.59
Hotplug status        : ONLINE
PCI Bar Address       : 0xb064000000
MSI                   : c9
PLLs locked           : Yes
PLLs Init Status      : PLL Initialized
PLLs Reset Status     : PLL Reset Skipped
Clock Status          : External (RP0)
Hardware ID           : |e08:3_e_2.0
Step 5

Verify that there are no TE alarms in the system.

Following is the list of TE alarms:

  • CLOCK_PORT_STATE_CHANGE

  • TIMING-PCI-ERROR

  • TIMING-LOAD-ERROR

  • TIMING-PLL-VAL-ERROR

  • CLK-PORT-STATUS-CHNG

  • TIMING-FPGA-SEU

  • TE-PORT-UNAVAILABLE

  • TIMING-ISOLATED-RACK
Step 6

Verify TE Port Topology, using the output of command show controllers timing controller te-port

  1. Verify that state of all the physical links. Link value should be is Good

    Note 

    If any of the TE Link is in No State, please check the physical connections.

  2. Verify that the Peer Rack is discovered as per the topology.

  3. From the FSYNC Mastership value, verify that only one rack converges as PRIMARY and remaining as SECONDARY.

    Note 

    FSYNC Mastership value should not be ISOLATED or SLAVE-READY or LISTENING or LEARNING.

  4. Verify from the TE state value, that all ports (TE0-E, TE1-E, TE0-W,and TE1-W) are in FORWARDING or MASTER or BACKUP or ALTERNATE state.

  5. TE State for primary rack should have value FORWARDING for all TE ports.

  6. TE State for secondary rack should have values MASTER – BACKUP – FORWARDING – FORWARDING or MASTER – BACKUP – ALTERNATE – ALTERNATE for TE0-E, TE1-E, TE0-W,and TE1-W ports respectively.

  7. Verify that Delay value is not zero.

Example:


RP/2/RP0:MC_FLT+4+1# show controllers timing controller te-port 
Thu Mar 22 11:43:01.307 IST

FSYNCDIR TE-Port Setting: Rack 0

FSYNC Mastership Rack 0: MASTER
          TE0-E           TE1-E          TE0-W          TE1-W 
TE state : FORWARDING     FORWARDING     FORWARDING     FORWARDING     
Rx Signal: No             No             No             No             
Link     : Good           Good           Good           Good           
PeerRack : 1              1              3              3              
PeerPort : TE0-W          TE1-W          TE0-E          TE1-E          
DELAY(ns): 240            240            235            240            


FSYNCDIR TE-Port Setting: Rack 1

FSYNC Mastership Rack 1: SLAVE
          TE0-E           TE1-E          TE0-W          TE1-W 
TE state : FORWARDING     FORWARDING     MASTER         BACKUP         
Rx Signal: No             No             Yes            Yes            
Link     : Good           Good           Good           Good           
PeerRack : 2              2              0              0              
PeerPort : TE0-W          TE1-W          TE0-E          TE1-E          
DELAY(ns): 235            240            240            240            
          

FSYNCDIR TE-Port Setting: Rack 2

FSYNC Mastership Rack 2: SLAVE
          TE0-E           TE1-E          TE0-W          TE1-W 
TE state : ALTERNATE      ALTERNATE      MASTER         BACKUP         
Rx Signal: Yes            Yes            Yes            Yes            
Link     : Good           Good           Good           Good           
PeerRack : 3              3              1              1              
PeerPort : TE0-W          TE1-W          TE0-E          TE1-E          
DELAY(ns): 240            235            240            240            


FSYNCDIR TE-Port Setting: Rack 3

FSYNC Mastership Rack 3: SLAVE
          TE0-E           TE1-E          TE0-W          TE1-W 
TE state : MASTER         BACKUP         ALTERNATE      ALTERNATE      
Rx Signal: Yes            Yes            Yes            Yes            
Link     : Good           Good           Good           Good           
PeerRack : 0              0              2              2              
PeerPort : TE0-W          TE1-W          TE0-E          TE1-E          
DELAY(ns): 235            240            240            235
Step 7

Verify Frequency Synchronization Selection Status, using the output of command show frequency synchronization selection

  1. Verify value for SYSTEM_T0_SEL. Following are valid output combinations:

    • If BITS or Frequency Synchronization source is configured then one of them should be in LOCKED state.

    • If BITS or Frequency Synchronization source is not configured then the Internal Clock can be in FREERUN or HOLDOVER state.

  2. Verify value for RACK<rackid>_SEL. Following are valid output combinations:

    • If <rackid> is Primary Rack, then it should have either BITS or Frequency Synchronization in LOCKED state or Internal in FREERUN or HOLDOVER state.

    • If <rackid> is Secondary Rack, then it should be LOCKED to TE port always..

Example:


RP/2/RP0:MC_FLT+4+1# show frequency synchronization selection 
Thu Mar 22 11:41:09.870 IST
Node 2/RP0:
==============
Selection point: SYSTEM_T0_SEL (6 inputs, 1 selected)
  Last programmed 17:05:34 ago, and selection made 17:04:19 ago
  Next selection points
    SPA scoped    : None
    Node scoped   : SYSTEM_T4_SEL
    Chassis scoped: None
    Router scoped : None
  Uses frequency selection
  Used for local line interface output
  S  Input                     Last Selection Point         QL  Pri  Status
  == ========================  ========================  =====  ===  ===========
  1  Rack0-Bits0-In            2/RP0 RACK0_SEL 1           PRC    9  Locked     
     Rack2-Bits0-In            2/RP0 RACK2_SEL 3         SSU-B    9  Available  
     Internal0 [2/RP0]         2/RP0 RACK0_SEL 2           SEC  255  Available  
     Internal0 [2/RP0]         2/RP0 RACK1_SEL 3           SEC  255  Available  
     Internal0 [2/RP0]         2/RP0 RACK2_SEL 4           SEC  255  Available  
     Internal0 [2/RP0]         2/RP0 RACK3_SEL 3           SEC  255  Available  

Selection point: SYSTEM_T4_SEL (2 inputs, 1 selected)
  Last programmed 17:06:27 ago, and selection made 17:04:19 ago
  Next selection points
    SPA scoped    : None
    Node scoped   : None
    Chassis scoped: None
    Router scoped : None
  Uses frequency selection
  Used for local clock interface output
  S  Input                     Last Selection Point         QL  Pri  Status
  == ========================  ========================  =====  ===  ===========
  1  Rack0-Bits0-In            2/RP0 SYSTEM_T0_SEL 1       PRC    9  Locked     
     Internal0 [2/RP0]         n/a                         SEC  255  Available  

Selection point: RACK0_SEL (2 inputs, 2 selected)
  Last programmed 17:06:27 ago, and selection made 17:04:19 ago
  Next selection points
    SPA scoped    : None
    Node scoped   : SYSTEM_T0_SEL
    Chassis scoped: None
    Router scoped : None
  Uses frequency selection
  S  Input                     Last Selection Point         QL  Pri  Status
  == ========================  ========================  =====  ===  ===========
  1  Rack0-Bits0-In            n/a                         PRC    9  Locked     
  2  Internal0 [2/RP0]         n/a                         SEC  255  Available  
          
Selection point: RACK1_SEL (3 inputs, 1 selected)
  Last programmed 17:05:42 ago, and selection made 17:04:59 ago
  Next selection points
    SPA scoped    : None
    Node scoped   : SYSTEM_T0_SEL
    Chassis scoped: None
    Router scoped : None
  Uses frequency selection
  S  Input                     Last Selection Point         QL  Pri  Status
  == ========================  ========================  =====  ===  ===========
  3  Internal0 [2/RP0]         n/a                         SEC  255  Available  
     1/TE0-W                   n/a                         PRC  100  Locked     
     1/TE1-W                   n/a                         PRC  100  Unmonitored

Selection point: RACK2_SEL (4 inputs, 2 selected)
  Last programmed 17:05:36 ago, and selection made 17:04:24 ago
  Next selection points
    SPA scoped    : None
    Node scoped   : SYSTEM_T0_SEL
    Chassis scoped: None
    Router scoped : None
  Uses frequency selection
  S  Input                     Last Selection Point         QL  Pri  Status
  == ========================  ========================  =====  ===  ===========
  3  Rack2-Bits0-In            n/a                       SSU-B    9  Available  
  4  Internal0 [2/RP0]         n/a                         SEC  255  Available  
     2/TE0-W                   n/a                         PRC  100  Locked     
     2/TE1-W                   n/a                         PRC  100  Unmonitored

Selection point: RACK3_SEL (3 inputs, 1 selected)
  Last programmed 17:05:39 ago, and selection made 17:04:45 ago
  Next selection points
    SPA scoped    : None
    Node scoped   : SYSTEM_T0_SEL
    Chassis scoped: None
    Router scoped : None
  Uses frequency selection
  S  Input                     Last Selection Point         QL  Pri  Status
  == ========================  ========================  =====  ===  ===========
  3  Internal0 [2/RP0]         n/a                         SEC  255  Available  
     3/TE0-E                   n/a                         PRC  100  Locked     
     3/TE1-E                   n/a                         PRC  100  Unmonitored
Step 8

Verify the clock data table for the BITS-In or TE interfaces, using the output of command show frequency synchronization clock-interfaces

  1. Verify that for primary or backup TE Ports, the INPUT should in UP state with proper QL Value.

  2. Verify that QL Value (Quality) is not DNU.

Example:


RP/2/RP0:MC_FLT+4+1#show frequency synchronization clock-interfaces       
Thu Mar 22 12:27:11.744 IST
  Clock interface Rack0-Bits0-In (Up - BITS 2M)
    Assigned as input for selection
    Wait-to-restore time 5 minutes
    SSM supported
    Input:
      Up
      Configured QL: Opt-I/PRC
      Effective QL: Opt-I/PRC, Priority: 9, Time-of-day Priority 100
      Supports frequency
    Output is disabled
  Next selection points: RACK0_SEL

  Clock interface Rack0-Bits0-Out (Unknown state)
    Wait-to-restore time 5 minutes
    SSM supported and enabled
    Input is disabled
    Output:
      Selected source: Rack0-Bits0-In
      Selected source QL: Opt-I/PRC
      Effective QL: Opt-I/PRC
  Next selection points: None

  Clock interface Rack0-Bits1-In (Unknown state)
    Wait-to-restore time 5 minutes
    SSM supported and enabled
    Input:
      Down - not assigned for selection
      Last received QL: None
      Supports frequency
    Output is disabled
  Next selection points: RACK0_SEL

  Clock interface Rack0-Bits1-Out (Unknown state)
    Wait-to-restore time 5 minutes
    SSM supported and enabled
    Input is disabled
    Output:
      Selected source: Rack0-Bits0-In
      Selected source QL: Opt-I/PRC
      Effective QL: Opt-I/PRC
  Next selection points: None

  Clock interface 0/TE0-E (Up - Inter-Chassis Sync)
    Wait-to-restore time 5 minutes
    SSM supported and enabled
    Input is disabled
    Output:
      Selected source: Rack0-Bits0-In
      Selected source QL: Opt-I/PRC
      Effective QL: Opt-I/PRC
  Next selection points: None

  Clock interface 0/TE1-E (Up - Inter-Chassis Sync)
    Wait-to-restore time 5 minutes
    SSM supported and enabled
    Input is disabled
    Output:
      Selected source: Rack0-Bits0-In
      Selected source QL: Opt-I/PRC
      Effective QL: Opt-I/PRC
  Next selection points: None

  Clock interface 0/TE0-W (Up - Inter-Chassis Sync)
    Wait-to-restore time 5 minutes
    SSM supported and enabled
    Input is disabled
    Output:
      Selected source: Rack0-Bits0-In
      Selected source QL: Opt-I/PRC
      Effective QL: Opt-I/PRC
  Next selection points: None

  Clock interface 0/TE1-W (Up - Inter-Chassis Sync)
    Wait-to-restore time 5 minutes
    SSM supported and enabled
    Input is disabled
    Output:
      Selected source: Rack0-Bits0-In
      Selected source QL: Opt-I/PRC
      Effective QL: Opt-I/PRC
  Next selection points: None
Step 9

Verify SYNCE_IN interface status using following substeps:

  1. Verify that the SYNCE interfaces are not in Operationally Down State using command show frequency synchronization interfaces brief

    Example:

    
RP/0/RP0:MC_OTN#show frequency synchronization interfaces brief 
Thu Mar 22 14:42:52.032 IST
Flags:  > - Up                D - Down              S - Assigned for selection
        d - SSM Disabled      x - Peer timed out    i - Init state
        s - Output squelched
Fl   Interface                QLrcv QLuse Pri QLsnd Output driven by
==== ======================== ===== ===== === ===== ========================
>    TenGigE0/9/0/2           DNU   n/a   100 PRC   Rack2-Bits0-In          
>S   TenGigE0/9/0/8           PRC   PRC   200 PRC   Rack2-Bits0-In          
>S   TenGigE2/4/0/2           SSU-A SSU-A 100 PRC   Rack2-Bits0-In          
>S   FortyGigE2/15/0/6        PRC   PRC    10 PRC   Rack2-Bits0-In

  2. Verify that the SSM packets are being sent and received using command show frequency synchronization interfaces

    Example:

    
RP/0/RP0:MC_OTN# show frequency synchronization interfaces 
Thu Mar 22 14:45:46.452 IST
Interface TenGigE0/9/0/2 (up)
  Wait-to-restore time 5 minutes
  SSM Enabled
    Peer Up for 02:24:29, last SSM received 0.717s ago
    Peer has come up 1 times and timed out 0 times
    ESMC SSMs        Total  Information     Event    DNU/DUS
      Sent:           8672         8671         1          0
      Received:       8672         8668         4       8645
  Input:
    Down - not assigned for selection
    Supports frequency
  Output:
    Selected source: Rack2-Bits0-In
    Selected source QL: Opt-I/PRC
    Effective QL: Opt-I/PRC
  Next selection points: RACK0_SEL