Overview

Consider the following topology:

Figure 1. Sample Topology Depicting Converged Packet-based Fronthaul Scenario

In this topology, a centralized RAN is deployed for Fronthaul and is applicable to both, 4G LTE and 5G NR Radio Systems. The broadband unit (BBU) is used for CPRI packet generation as well as the Primary Reference Clock for timing and synchronization.

Port-3 of the BBU is configured as the CPRI Master, while the CPRI Interface of the Cisco NCS 540-FH-CSR-SYS (R4) (GE0/0/0/7) connected to it, is configured as the CPRI Slave. Port-3 is also configured as the Primary Reference clock or Source Clock to derive the time. The clock is transmitted over the Ethernet interfaces towards the core devices. Port-1 on the radio unit (RU) reads the clock from the CPRI frame it receives over the CPRI Interface, which in turn is connected to another Cisco N540-FH-CSR-SYS (R4) on the other side.

The CPRI traffic is carried over by a P2P Xconnect running between the two Cisco NCS 540-FH routers (R3 and R4) that create the packetized network to carry the Radio Traffic. The core network runs the IGP and SR-MPLS. The signaling protocol for L2VPN is targeted-LDP.

There are two modes of operation:

  1. Type 0 or Tunneling Mode: This mode works as a simple Ethernet tunnel. It does not remove any line coding bits and cannot interpret any special characters.

  2. Type 1: This mode is Line Coding aware. It removes the line coding bits and saves approximately 20% of the bandwidth.

Prerequisites

The following prerequisites are required to bring up the underlay for the CPRI circuit. Underlay is the transport (IGP+SR-MPLS) that should be built to bring up the CRPI services. To do this, ISIS, SR-MPLS, and TI-LFA are configured under IS-IS.

  1. On the router connected to the baseband unit (R3):

    router isis Core
 is-type level-2-only
 net 49.0000.0000.0001.00
 segment-routing global-block 16000 23999
 address-family ipv4 unicast
  metric-style wide
  router-id Loopback0
  segment-routing mpls sr-prefer
 ! 
 interface Loopback0
  address-family ipv4 unicast
   prefix-sid index 1001
  !
 !
 interface TenGigE0/0/0/15
  point-to-point
  address-family ipv4 unicast
   fast-reroute per-prefix
   fast-reroute per-prefix ti-lfa
  !
 !
 interface TenGigE0/0/0/17
  point-to-point
  address-family ipv4 unicast
   fast-reroute per-prefix
   fast-reroute per-prefix ti-lfa
  !

  2. On the router (R4) connected to the radio unit:

    router isis Core
 is-type level-2-only
 net 49.0000.0000.0004.00
 segment-routing global-block 16000 23999
 address-family ipv4 unicast
  metric-style wide
  router-id Loopback0
  segment-routing mpls sr-prefer
 !
 interface Loopback0
  address-family ipv4 unicast
   prefix-sid index 1004
  !
 !
 interface TenGigE0/0/0/14
  point-to-point
  address-family ipv4 unicast
   fast-reroute per-prefix
   fast-reroute per-prefix ti-lfa
  !
 !
 interface TenGigE0/0/0/16
  point-to-point
  address-family ipv4 unicast
   fast-reroute per-prefix
   fast-reroute per-prefix ti-lfa
  !

  3. On R1 (connected to R3):

    router isis Core
 is-type level-2-only
 net 49.0000.0000.0002.00
 segment-routing global-block 16000 23999
 address-family ipv4 unicast
  metric-style wide
  router-id Loopback0
  segment-routing mpls sr-prefer
 !
 interface Loopback0
  address-family ipv4 unicast
   prefix-sid index 1002
  !
 !
 interface TenGigE0/0/0/20
  point-to-point
  address-family ipv4 unicast
   fast-reroute per-prefix
   fast-reroute per-prefix ti-lfa
  !
 !
 interface TenGigE0/0/0/21
  point-to-point
  address-family ipv4 unicast
   fast-reroute per-prefix
   fast-reroute per-prefix ti-lfa

  4. On R2 (connected to R4):

    router isis Core
 is-type level-2-only
 net 49.0000.0000.0003.00
 address-family ipv4 unicast
  metric-style wide
  router-id Loopback0
  segment-routing mpls sr-prefer
 !
 interface Loopback0
  address-family ipv4 unicast
   prefix-sid index 1003
  !
 !
 interface TenGigE0/0/0/20
  point-to-point
  address-family ipv4 unicast
   fast-reroute per-prefix
   fast-reroute per-prefix ti-lfa
  !
 !
 interface TenGigE0/0/0/21
  point-to-point
  address-family ipv4 unicast
   fast-reroute per-prefix
   fast-reroute per-prefix ti-lfa

Connecting the Port to the Baseband Unit

  1. Enable controller optics on which CPRI role and CPRI Option Rate are enabled. The following example shows the Proxy Slave port connected to the BBU/DU. The CPRI Option configured is 3 and the rate therefore is 2.457 Gbps. 

    controller Optics0/0/0/7
 port-mode cpri role slave datarate-list 2457  
!

  2. Define the RoE profile with:

    1. Ethernet link

      roe
 ethlink 1
  destination-MAC aaaa.bbbb.cccc
  source-MAC 1234.5678.abcd

      Ensure the destma matches the srcmac and the srcmac matches the destmac of the RoE Profile Ethernet link on the Cisco N540-FH-CSR-SYS facing the RU.

    2. Order-Info – Both, sequence number and timestamp must be used.

    3. Map-Flow-ID – Should match the Demap-flow-ID on the N540-FH-CSR-SYS facing the RU

    4. Mapper-Type – Should point to Structure Agnostic Type-0 (Simple Tunneling Mode)

    5. Retimer-Buf – Size of the retimer buffer in De-mapper


      Note

      It is recommended that you do not change the default values.

    6. DeMap-Flow-ID – Should match the Map-flow-ID on the N540-FH-CSR-SYS facing the RU

    7. Idle-Frame-Size -Size of the idle-frame buffer configured for the port


      Note

      It is recommended that you do not change the default values.

    8. Idle-Frame-Thresh – Number of consecutive idle frames to be sent before which link will be brought down


      Note

      It is recommended that you do not change the default values.

    9. By default, hyperframe number synchronization (HFN Sync) for CPRI is enabled only for Structure Agnostic tunneling mode (Type-0) mapper type config in non auto-negotiation CPRI port. You can disable this configuration using the following command in RoE Profile. 

      roe
profile 108
hfn-sync-disable

  3. Associate an Ethernet link to the RoE Profile.

    roe
profile 1
  ethlink 1
  order-info SEQ-NUM-AND-TIMESTAMP
  map-flow-id 123
  mapper-type STR-AGN-TYPE0
  retimer-buf-size 0
  demap-flow-id 123
  idle-frame-thresh 0
 !

  4. Attach the RoE Mapper profile to the CPRI controller.

    controller CPRI0/0/0/7
 attach-roe-profile 1
!

  5. Enable Synchronous Ethernet (SyncE) globally. Frequency and Timing Synchronization distribute precision frequency around a network. Specify the Quality Level and ensure the same level is specified on the Interface. 

    frequency synchronization
 quality itu-t option 1

  6. Enable SyncE on CPRI over Ethernet (CPRIoE) interface to retrieve the timing from BBU/DU.

    interface CPRIoE0/0/0/7
 mtu 9600
 frequency synchronization
  selection input
  wait-to-restore 0
  ssm disable
quality receive exact itu-t option 1 PRC

    Note

    For clock design, see the Best Practices Guide for Cisco Converged Fronthaul Systems.

  7. On R3, enable MTU and layer 2 transport on the CPRIoE interface.

    interface CPRIoE0/0/0/7
 mtu 9600
 l2transport

Connecting Port to Radio Unit

  1. Enable controller optics on which CPRI role and CPRI Option Rate are enabled. The following example shows the Proxy Master port connected to the Radio Unit. The CPRI Option configured is 3 and the rate therefore is 2.457 Gbps. 

    controller Optics0/0/0/7
 port-mode cpri role master datarate-list 2457

  2. Define the RoE profile with:

    1. Ethernet link

      roe
 ethlink 1
  destination-MAC 1234.5678.abcd 
  source-MAC aaaa.bbbb.cccc

      Ensure the destma matches the srcmac and the srcmac matches the destmac of the RoE Profile Ethernet link on the Cisco N540-FH-CSR-SYS facing the BBU/DU.

    2. Order-Info – Both, sequence number and timestamp must be used.

    3. Map-Flow-ID – Should match the Demap-flow-ID on the N540-FH-CSR-SYS facing the RU

    4. Mapper-Type – Should point to Structure Agnostic Type-0 (Simple Tunneling Mode)

    5. Retimer-Buf – Size of the retimer buffer in De-mapper


      Note

      It is recommended that you do not change the default values.

    6. DeMap-Flow-ID – Should match the Map-flow-ID on the N540-FH-CSR-SYS facing the RU

    7. Idle-Frame-Size -Size of the idle-frame buffer configured for the port


      Note

      It is recommended that you do not change the default values.

    8. Idle-Frame-Thresh – Number of consecutive idle frames to be sent before which link will be brought down


      Note

      It is recommended that you do not change the default values.

    9. By default, hyperframe number synchronization (HFN Sync) for CPRI is enabled only for Structure Agnostic tunneling mode (Type-0) mapper type config in non auto-negotiation CPRI port. You can disable this configuration using the following command in RoE Profile. 

      roe
profile 108
hfn-sync-disable

  3. Associate an Ethernet link to the RoE Profile.

    roe
profile 1
  ethlink 1
  order-info SEQ-NUM-AND-TIMESTAMP
  map-flow-id 123
  mapper-type STR-AGN-TYPE0
  retimer-buf-size 0
  demap-flow-id 123
  idle-frame-thresh 0
 !

  4. Attach the RoE Mapper profile to the CPRI controller.

    controller CPRI0/0/0/7
 attach-roe-profile 1
!

  5. Enable Synchronous Ethernet on this interface to retrieve the timing from Core Routers (that retrieved the timing from the N540-FH-CSR-SYS facing the BBU/DU) 

    interface TenGigE0/0/0/14
 description "Connected to BB1 || SR-MPLS/ISIS Core || TenGigE0/0/0/20"
 mtu 9600
 ipv4 address 10.1.4.1 255.255.255.0
 load-interval 30
 frequency synchronization
  selection input
  wait-to-restore 0
  ssm disable
  quality receive exact itu-t option 1 PRC

interface TenGigE0/0/0/16
 description "Connected to BB2 || SR-MPLS/ISIS Core || TenGigE0/0/0/20"
 mtu 9600
 ipv4 address 10.1.5.1 255.255.255.0
 load-interval 30
 frequency synchronization
  selection input
  wait-to-restore 0
  ssm disable
  quality receive exact itu-t option 1 PRC

    Note

    For clock design, see the Best Practices Guide for Cisco Converged Fronthaul Systems.

  6. On R4, enable MTU and layer 2 transport on the CPRIoE interface

    interface CPRIoE0/0/0/7
 mtu 9600
 l2transport

Enabling EVPN

EVPN provides an end-to-end layer 2 connection.

Prerequisites

  • BGP EVPN Address Family must be configured on R3 and R4.

  1. On the router (R3) connected to the baseband unit:

    router bgp 108
 bgp router-id 11.11.11.11
 address-family l2vpn evpn
 !
 neighbor 44.44.44.44
  remote-as 108
  update-source Loopback0
  address-family l2vpn evpn
  !

  2. On the router (R4) connected to the radio unit:

    router bgp 108
 bgp router-id 44.44.44.44
 address-family l2vpn evpn
 !
 neighbor 11.11.11.11
  remote-as 108
  update-source Loopback0
  address-family l2vpn evpn
  !

Configuring EVPN VPWS

  1. On R3:

    evpn
 evi 108
 !
l2vpn
xconnect group evpn-vpws108
  p2p evpn108
   interface CPRIoE0/0/0/7
   neighbor evpn evi 108 service 108

  2. On R4:

    evpn
 evi 108
 !
l2vpn
xconnect group evpn-vpws108
  p2p evpn108
   interface CPRIoE0/0/0/7
   neighbor evpn evi 108 service 108

For more einformation on configuring EVPN, see the L2VPN and Ethernet Services Configuration Guide for Cisco NCS 540 Series Routers.

Configuring PTP

Adding PTP to a network can compensate for latency and delay problems by correctly adjusting device clocks so that they stay synchronized with one another.

To configure PTP globally:

ptp
 clock
  domain 24
  profile g.8275.1 clock-type T-GM
  timescale PTP
  time-source GPS
  clock-class 6
 !
 profile slave
  transport ethernet
  sync frequency 16
  announce frequency 8
  delay-request frequency 16
 !
 profile master
  multicast target-address ethernet 01-1B-19-00-00-00
  transport ethernet
  port state master-only
  sync frequency 16
  clock operation one-step
  announce frequency 8
  delay-request frequency 16
 !
 physical-layer-frequency
!

To configure PTP on every interface:

interface HundredGigE0/0/0/26
 description From_T-GM
 ptp
  profile slave
  multicast target-address ethernet 01-1B-19-00-00-00
  transport ethernet
  port state slave-only
  local-priority 10
 !
 frequency synchronization
  selection input
  priority 1
  wait-to-restore 0
  ssm disable
  quality receive exact itu-t option 1 PRC
 !
!
interface HundredGigE0/0/0/27
 description To_T-BC Felidae Proxy-Master
 ptp
  profile master
  multicast target-address ethernet 01-1B-19-00-00-00
  transport ethernet
  port state master-only
  sync frequency 64
  clock operation one-step
  announce frequency 16
  delay-request frequency 64
 !
 frequency synchronization
wait-to-restore 0

For more information on configuring PTP, see the:.

  • Best Practices Guide for Cisco Converged Fronthaul Systems

  • Configuring PTP chapter in the Network Synchronization Configuration Guide for Cisco NCS 540 Routers.

Configuring QoS

QoS is configured to improve performance for critical network traffic.

To configure QoS globally:

class-map match-any exp5
 match mpls experimental topmost 5 
 end-class-map   
! 
policy-map Radio_Interface_Xhaul
 class class-default
  set traffic-class 7
  set qos-group 5
 ! 
 end-policy-map
!      
policy-map Core_Interface_Xhaul
 class exp5
  set traffic-class 7
 ! 
 end-policy-map
!

To configure QoS on radio interfaces:

interface CPRIoE0/0/0/0
 mtu 9600
 load-interval 30
 !
 l2transport
  service-policy input Radio_Interface_Xhaul

For more information on configuring QoS, see the:.

  • Best Practices Guide for Cisco Converged Fronthaul Systems

  • Modular QoS Configuration Guide for Cisco NCS 540 Routers.

Configuring Low Latency Queuing for Specific QoS Flows

Table 1. Feature History Table

Feature Name

Release Information

Feature Description

Low Latency Queuing for Specific QoS Flows

Release 7.3.2

This feature allows you to configure low latency or priority for specific QoS flows at the class-level within a policy map. It enables delay-sensitive data to be prioritized and transmitted before other packets in other queues.

This feature allows you to configure low latency or priority for specific QoS flows at the class-level within a policy map. It enables delay-sensitive data to be prioritized and transmitted before other packets in other queues.

Low Latency Queuing (LLQ) is achieved by enabling low delay IPS profile for the designated Virtual Output Queuing (VOQ).

Configuring Low Latency Queuing for Specific QoS Flows

Ingress Class Map Configurations

class-map match-any inDSCP_CS6
match dscp cs6
end-class-map
!
class-map match-any inDSCP_CS7
match dscp cs7
end-class-map
!

Egress Class Map Configuration

class-map match-any outCM6
match traffic-class 6
end-class-map
!
class-map match-any outCM7
match traffic-class 7
end-class-map
!

Ingress Policy Map Configurations

policy-map Ingress
class inDSCP_CS6
set traffic-class 6
!
class inDSCP_CS7
set traffic-class 7
!
class class-default
!
end-policy-map
!

Egress Policy Map Configuration

policy-map Egress
class outCM6
priority level 2
low-latency-profile

!
class outCM7
priority level 1
low-latency-profile
!

Verification

Router# show controllers npu voq-usage interface all instance all location all
Sat Jul 24 23:33:07.038 UTC

-------------------------------------------------------------------
Node ID: 0/RP0/CPU0
Intf         Intf     NPU NPU  PP   Sys   VOQ   Flow   VOQ    Port 
name         handle    #  core Port Port  base  base   port   speed
             (hex)                                     type        
----------------------------------------------------------------------
Hu0/0/0/24   3c000048   0   0    9     9   1032   5384 local   100G
Hu0/0/0/25   3c0000a8   0   0   21    21   1040   5392 local   100G

RP/0/RP0/CPU0:Lion#show  controllers fia   diagshell 0 "diag cosq voq id=1046 det=1 " location 0/RP0/CPU0 
Sat Jul 24 23:34:18.904 UTC

Node ID: 0/RP0/CPU0

R/S/I: 0/0/0 

Core 0:
Basic info
        Q type: voq
        num cos: 8, cosq class: 6
        Base queue id: 1040, base queue gport: 0x24000410
Credit request type: BCM_COSQ_DELAY_TOLERANCE_100G_LOW_DELAY Adjusted for low delay 100Gb ports
Watchdog enable in common status message mode
        Is queue in credit watchdog queue range:True
        Credit watchdog message time: 0
        Delete queue time: 512
        Backoff enter queue credit balance threshold: 98304
        Backoff exit queue credit balance threshold: 98304
        Backlog enter queue credit balance threshold: 7168
        Backlog exit queue credit balance threshold: 7168
        Empty queue satisfied credit balance threshold: 6720
        Max empty queue credit balance threshold: 11200
        Exceed max empty queue credit balance threshold: 1
        Off-To-Slow credit balance threshold: -11264
        Off-To-Normal credit balance threshold: -11264
        Slow-To-Normal credit balance threshold: -11264
        Normal-To-Slow credit balance threshold: -11264
        Delay Tolerance is OCB only

Limitations

  • No support on logical interfaces and sub-interfaces.

  • Device on which low latency is enabled, throughput may be degraded.

  • We recommend that you enable only one high priority traffic class (TC) per port for low latency. Enabling multiple TC with low latency may result in burst.


Note
Configurations are restricted on any TCs, but it is recommended that you enable LLQ on only one TC.

Verifying the Fronthaul Configuration

Verifying MPLS Transport

On R3:


RP/0/RP0/CPU0:R3# show isis adjacency
Mon Jan 20 17:30:14.923 UTC
IS-IS Core Level-2 adjacencies:
System Id      Interface                SNPA           State Hold Changed  NSF IPv4 IPv6
                                                                               BFD  BFD
BB1            Te0/0/0/15               *PtoP*         Up    23   6d00h    Yes None None
BB2            Te0/0/0/17               *PtoP*         Up    21   5d23h    Yes None None

RP/0/RP0/CPU0:R3#show isis segment-routing label table
Mon Jan 20 17:30:30.242 UTC
IS-IS Core IS Label Table
Label         Prefix/Interface
----------    ----------------
17001         Loopback0
17002         22.22.22.22/32
17003         33.33.33.33/32
17004         44.44.44.44/32

RP/0/RP0/CPU0:R3# show mpls label table
Mon Jan 20 17:31:12.590 UTC
Table Label   Owner                           State  Rewrite
----- ------- ------------------------------- ------ -------
0     0       LSD(A)                          InUse  Yes
0     1       LSD(A)                          InUse  Yes
0     2       LSD(A)                          InUse  Yes
0     13      LSD(A)                          InUse  Yes
0     15000   LSD(A)                          InUse  No
0     16000   ISIS(A):Core                    InUse  No
0     24000   ISIS(A):Core                    InUse  Yes
0     24001   ISIS(A):Core                    InUse  Yes
0     24005   L2VPN(A)                        InUse  Yes
0     24007   ISIS(A):Core                    InUse  Yes
0     24008   ISIS(A):Core                    InUse  Yes
RP/0/RP0/CPU0:R3#


RP/0/RP0/CPU0:R3# show isis fast-reroute sr-only
Mon Jan 20 17:31:44.919 UTC
IS-IS Core IPv4 Unicast FRR backups
Codes: L1 - level 1, L2 - level 2, ia - interarea (leaked into level 1)
       df - level 1 default (closest attached router), su - summary null
       C - connected, S - static, R - RIP, B - BGP, O - OSPF
       E - EIGRP, A - access/subscriber, M - mobile, a - application
       i - IS-IS (redistributed from another instance)
       D - Downstream, LC - Line card disjoint, NP - Node protecting
       P - Primary path, SRLG - SRLG disjoint, TM - Total metric via backup
Maximum parallel path count: 8
L2 22.22.22.22/32 [20/115]
     via 10.1.1.2, TenGigE0/0/0/15, BB1, SRGB Base: 16000, Weight: 0
       Backup path: TI-LFA (link), via 10.1.2.2, TenGigE0/0/0/17 BB2, SRGB Base: 16000, Weight: 0, Metric: 40
         P node: JAG2.00 [44.44.44.44], Label: 17004
         Prefix label: 17002
         Backup-src: BB1.00
L2 33.33.33.33/32 [20/115]
     via 10.1.2.2, TenGigE0/0/0/17, BB2, SRGB Base: 16000, Weight: 0
       Backup path: TI-LFA (link), via 10.1.1.2, TenGigE0/0/0/15 BB1, SRGB Base: 16000, Weight: 0, Metric: 40
         P node: JAG2.00 [44.44.44.44], Label: 17004
         Prefix label: 17003
         Backup-src: BB2.00
L2 44.44.44.44/32 [30/115]
     via 10.1.1.2, TenGigE0/0/0/15, BB1, SRGB Base: 16000, Weight: 0
       Backup path: LFA, via 10.1.2.2, TenGigE0/0/0/17, BB2, SRGB Base: 16000, Weight: 0, Metric: 30
     via 10.1.2.2, TenGigE0/0/0/17, BB2, SRGB Base: 16000, Weight: 0
       Backup path: LFA, via 10.1.1.2, TenGigE0/0/0/15, BB1, SRGB Base: 16000, Weight: 0, Metric: 30
RP/0/RP0/CPU0:R3#

On R4:

RP/0/RP0/CPU0:R4# show isis adjacency
Mon Jan 20 17:34:46.196 UTC
IS-IS Core Level-2 adjacencies:
System Id      Interface                SNPA           State Hold Changed  NSF IPv4 IPv6
                                                                               BFD  BFD
BB1            Te0/0/0/14               *PtoP*         Up    25   6d00h    Yes None None
BB2            Te0/0/0/16               *PtoP*         Up    21   5d23h    Yes None None
Total adjacency count: 2

RP/0/RP0/CPU0:R4# show isis segment-routing label table
Mon Jan 20 17:34:50.834 UTC
IS-IS Core IS Label Table
Label         Prefix/Interface
----------    ----------------
17001         11.11.11.11/32
17002         22.22.22.22/32
17003         33.33.33.33/32
17004         Loopback0


RP/0/RP0/CPU0:R4# show mpls label table
Mon Jan 20 17:34:56.081 UTC
Table Label   Owner                           State  Rewrite
----- ------- ------------------------------- ------ -------
0     0       LSD(A)                          InUse  Yes
0     1       LSD(A)                          InUse  Yes
0     2       LSD(A)                          InUse  Yes
0     13      LSD(A)                          InUse  Yes
0     15000   LSD(A)                          InUse  No
0     16000   ISIS(A):Core                    InUse  No
0     24002   ISIS(A):Core                    InUse  Yes
0     24003   ISIS(A):Core                    InUse  Yes
0     24004   L2VPN(A)                        InUse  Yes
0     24008   ISIS(A):Core                    InUse  Yes
0     24009   ISIS(A):Core                    InUse  Yes

RP/0/RP0/CPU0:R4# show isis fast-reroute sr-only
Mon Jan 20 17:35:00.055 UTC
IS-IS Core IPv4 Unicast FRR backups
Codes: L1 - level 1, L2 - level 2, ia - interarea (leaked into level 1)
       df - level 1 default (closest attached router), su - summary null
       C - connected, S - static, R - RIP, B - BGP, O - OSPF
       E - EIGRP, A - access/subscriber, M - mobile, a - application
       i - IS-IS (redistributed from another instance)
       D - Downstream, LC - Line card disjoint, NP - Node protecting
       P - Primary path, SRLG - SRLG disjoint, TM - Total metric via backup
Maximum parallel path count: 8
L2 11.11.11.11/32 [30/115]
     via 10.1.4.2, TenGigE0/0/0/14, BB1, SRGB Base: 16000, Weight: 0
       Backup path: LFA, via 10.1.5.2, TenGigE0/0/0/16, BB2, SRGB Base: 16000, Weight: 0, Metric: 30
     via 10.1.5.2, TenGigE0/0/0/16, BB2, SRGB Base: 16000, Weight: 0
       Backup path: LFA, via 10.1.4.2, TenGigE0/0/0/14, BB1, SRGB Base: 16000, Weight: 0, Metric: 30
L2 22.22.22.22/32 [20/115]
     via 10.1.4.2, TenGigE0/0/0/14, BB1, SRGB Base: 16000, Weight: 0
       Backup path: TI-LFA (link), via 10.1.5.2, TenGigE0/0/0/16 BB2, SRGB Base: 16000, Weight: 0, Metric: 40
         P node: R3.00 [11.11.11.11], Label: 17001
         Prefix label: 17002
         Backup-src: BB1.00
L2 33.33.33.33/32 [20/115]
     via 10.1.5.2, TenGigE0/0/0/16, BB2, SRGB Base: 16000, Weight: 0
       Backup path: TI-LFA (link), via 10.1.4.2, TenGigE0/0/0/14 BB1, SRGB Base: 16000, Weight: 0, Metric: 40
         P node: R3.00 [11.11.11.11], Label: 17001
         Prefix label: 17003
         Backup-src: BB2.00
RP/0/RP0/CPU0:R4#

Verifying Services (Xconnect, EVPN Status, Packets Received and Transmitted)

On R3:

RP/0/RP0/CPU0:R3# show l2vpn xconnect pw-id 2 detail
Mon Jan 20 17:39:41.124 UTC
Group 2, XC 2, state is up; Interworking none
  AC: CPRIoE0/0/0/7, state is up
    Type Ethernet
    MTU 9586; XC ID 0x1; interworking none
    Statistics:
      packets: received 61963291278, sent 61963291274
      bytes: received 160237071244908, sent 160237071234564
  PW: neighbor 44.44.44.44, PW ID 2, state is up ( established )
    PW class not set, XC ID 0xc0000002
    Encapsulation MPLS, protocol LDP
    Source address 11.11.11.11
    PW type Ethernet, control word disabled, interworking none
    PW backup disable delay 0 sec
    Sequencing not set
    Ignore MTU mismatch: Disabled
    Transmit MTU zero: Disabled
    LSP : Up
    PW Status TLV in use
      MPLS         Local                          Remote
      ------------ ------------------------------ -----------------------------
      Label        24005                          24004
      Group ID     0x3c040f8                      0x3c040f8
      Interface    CPRIoE0/0/0/7                  CPRIoE0/0/0/7
      MTU          9586                           9586
      Control word disabled                       disabled
      PW type      Ethernet                       Ethernet
      VCCV CV type 0x2                            0x2
                   (LSP ping verification)        (LSP ping verification)
      VCCV CC type 0x6                            0x6
                   (router alert label)           (router alert label)
                   (TTL expiry)                   (TTL expiry)
      ------------ ------------------------------ -----------------------------
    Incoming Status (PW Status TLV):
      Status code: 0x0 (Up) in Notification message
    Outgoing Status (PW Status TLV):
      Status code: 0x0 (Up) in Notification message
    MIB cpwVcIndex: 3221225474
    Create time: 14/01/2020 16:39:29 (6d01h ago)
    Last time status changed: 14/01/2020 17:07:32 (6d00h ago)
    Last time PW went down: 14/01/2020 17:07:17 (6d00h ago)
    Statistics:
      packets: received 61963291274, sent 61963291278
      bytes: received 160237071234564, sent 160237071244908

On R4:

RP/0/RP0/CPU0:R4# show l2vpn xconnect pw-id 2 detail
Mon Jan 20 17:38:33.349 UTC
Group 2, XC 2, state is up; Interworking none
  AC: CPRIoE0/0/0/7, state is up
    Type Ethernet
    MTU 9586; XC ID 0x1; interworking none
    Statistics:
      packets: received 62828756739, sent 62427728902
      bytes: received 162475164927054, sent 161438106940572
  PW: neighbor 11.11.11.11, PW ID 2, state is up ( established )
    PW class not set, XC ID 0xc0000002
    Encapsulation MPLS, protocol LDP
    Source address 44.44.44.44
    PW type Ethernet, control word disabled, interworking none
    PW backup disable delay 0 sec
    Sequencing not set
    Ignore MTU mismatch: Disabled
    Transmit MTU zero: Disabled
    LSP : Up
    PW Status TLV in use
      MPLS         Local                          Remote
      ------------ ------------------------------ -----------------------------
      Label        24004                          24005
      Group ID     0x3c040f8                      0x3c040f8
      Interface    CPRIoE0/0/0/7                  CPRIoE0/0/0/7
      MTU          9586                           9586
      Control word disabled                       disabled
      PW type      Ethernet                       Ethernet
      VCCV CV type 0x2                            0x2
                   (LSP ping verification)        (LSP ping verification)
      VCCV CC type 0x6                            0x6
                   (router alert label)           (router alert label)
                   (TTL expiry)                   (TTL expiry)
      ------------ ------------------------------ -----------------------------
    Incoming Status (PW Status TLV):
      Status code: 0x0 (Up) in Notification message
    Outgoing Status (PW Status TLV):
      Status code: 0x0 (Up) in Notification message
    MIB cpwVcIndex: 3221225474
    Create time: 14/01/2020 16:09:10 (6d01h ago)
    Last time status changed: 14/01/2020 17:05:36 (6d00h ago)
    Last time PW went down: 14/01/2020 17:05:36 (6d00h ago)
    Statistics:
      packets: received 62427728902, sent 62828756739
      bytes: received 161438106940572, sent 162475164927054
RP/0/RP0/CPU0:R4# show l2vpn xconnect pw-id 2 detail
Mon Jan 20 17:38:33.349 UTC
Group 2, XC 2, state is up; Interworking none
  AC: CPRIoE0/0/0/7, state is up
    Type Ethernet
    MTU 9586; XC ID 0x1; interworking none
    Statistics:
      packets: received 62828756739, sent 62427728902
      bytes: received 162475164927054, sent 161438106940572
  PW: neighbor 11.11.11.11, PW ID 2, state is up ( established )
    PW class not set, XC ID 0xc0000002
    Encapsulation MPLS, protocol LDP
    Source address 44.44.44.44
    PW type Ethernet, control word disabled, interworking none
    PW backup disable delay 0 sec
    Sequencing not set
    Ignore MTU mismatch: Disabled
    Transmit MTU zero: Disabled
    LSP : Up
    PW Status TLV in use
      MPLS         Local                          Remote
      ------------ ------------------------------ -----------------------------
      Label        24004                          24005
      Group ID     0x3c040f8                      0x3c040f8
      Interface    CPRIoE0/0/0/7                  CPRIoE0/0/0/7
      MTU          9586                           9586
      Control word disabled                       disabled
      PW type      Ethernet                       Ethernet
      VCCV CV type 0x2                            0x2
                   (LSP ping verification)        (LSP ping verification)
      VCCV CC type 0x6                            0x6
                   (router alert label)           (router alert label)
                   (TTL expiry)                   (TTL expiry)
      ------------ ------------------------------ -----------------------------
    Incoming Status (PW Status TLV):
      Status code: 0x0 (Up) in Notification message
    Outgoing Status (PW Status TLV):
      Status code: 0x0 (Up) in Notification message
    MIB cpwVcIndex: 3221225474
    Create time: 14/01/2020 16:09:10 (6d01h ago)
    Last time status changed: 14/01/2020 17:05:36 (6d00h ago)
    Last time PW went down: 14/01/2020 17:05:36 (6d00h ago)
    Statistics:
      packets: received 62427728902, sent 62828756739
      bytes: received 161438106940572, sent 162475164927054

Verifying CPRI for Configured Rates

On R3:

RP/0/RP0/CPU0:R3# show controllers cpri 0/0/0/7
Mon Jan 20 17:42:45.369 UTC
Port CPRI0/0/0/7:
 Admin State: In Service
 Oper State: Up
 Loopback Mode: None
 Synchronization Mode: Slave
 L1 startup timer: 0ms
 Detected Alarms : LOF RAI
 CPRI Datarate:
 Rates:                   614MB     1228MB    2457MB    3072MB    4915MB    6144MB    9830MB    8110MB    10137MB  12165MB   24330MB
 Supported Rate List:     Y         Y         Y         Y         Y         Y         Y         -         Y         -         -
 Configured Rate List:    -         -         Y         -         -         -         -         -         -         -         -
 Negotiated Rate:         -         -         Y         -         -         -         -         -         -         -         -
 Last clearing of "show controllers CPRI" counters never
 ROE Status:
 Is roe configured:                     True
 Idle frame TCA:                        False
 Persistent stray frames detected:      False
 Persistent lost frames detected:       False
 Retimer Buffer Overrun Alarm:          False
 Retimer Buffer Underrun Alarm:         False



RP/0/RP0/CPU0:R3# show controllers cpRI 0/0/0/7 roe-stats
Tue Jan 21 15:52:14.341 UTC
 ROE Statistics:
 Received Frames                         = 19120060722
 Received Good Frames                    = 19120060723
 Out of Seq Frames Detected              = 0
 Dropped Frames                          = 0
        Stray Frames Detected            = 0
        Errored Length Packets           = 0
        Duplicate Frames Detected        = 0
        Out of Seq Frames Dropped        = 0

 Output Frames                           = 19120060723
 Idle Frames Sent                        = 0

 Retimer Buffer Overrrun Count           = 0
 Retimer Buffer Underrrun Count          = 0
 K Byte Mismatch                         = 0
 Packet Delay Variation :
         max/min/avg                     = 441 / 75 / 77 
 Idle Frame TCA                          = 0
 Received RoE LCV count:                 = 0

On R4:

RP/0/RP0/CPU0:R4# show controllers cpri 0/0/0/7
Mon Jan 20 17:41:48.951 UTC
Port CPRI0/0/0/7:
 Admin State: In Service
 Oper State: Up
 Loopback Mode: None
 Synchronization Mode: Master
 L1 startup timer: 0ms
 Detected Alarms : LOF RAI
 CPRI Datarate:
 Rates:                   614MB     1228MB    2457MB    3072MB    4915MB    6144MB    9830MB    8110MB    10137MB  12165MB   24330MB
 Supported Rate List:     Y         Y         Y         Y         Y         Y         Y         -         Y         -         -
 Configured Rate List:    -         -         Y         -         -         -         -         -         -         -         -
 Negotiated Rate:         -         -         Y         -         -         -         -         -         -         -         -
 Last clearing of "show controllers CPRI" counters never
 ROE Status:
 Is roe configured:                     True
 Idle frame TCA:                        False
 Persistent stray frames detected:      False
 Persistent lost frames detected:       False
 Retimer Buffer Overrun Alarm:          False
 Retimer Buffer Underrun Alarm:         False



RP/0/RP0/CPU0:R4# show controllers cpRI 0/0/0/7 roe-stats
Tue Jan 21 15:49:10.895 UTC
 ROE Statistics:
 Received Frames                         = 19515615537
 Received Good Frames                    = 19515615539
 Out of Seq Frames Detected              = 0
 Dropped Frames                          = 0
        Stray Frames Detected            = 0
        Errored Length Packets           = 0
        Duplicate Frames Detected        = 0
        Out of Seq Frames Dropped        = 0

 Output Frames                           = 19515615539
 Idle Frames Sent                        = 0

 Retimer Buffer Overrrun Count           = 0
 Retimer Buffer Underrrun Count          = 0
 K Byte Mismatch                         = 0
 Packet Delay Variation :
         max/min/avg                     = 437 / 72 / 73 
 Idle Frame TCA                          = 0
 Received RoE LCV count:                 = 0
RP/0/RP0/CPU0:R4#

Verifying PTP Configuration

On R3:

RP/0/RP0/CPU0:R3# show frequency synchronization selection
Mon Jan 20 17:17:14.283 UTC
Node 0/RP0/CPU0:
==============
Selection point: T0-SEL-B (2 inputs, 1 selected)
  Last programmed 6d00h ago, and selection made 02:54:01 ago
  Next selection points
    SPA scoped    : None
    Node scoped   : CHASSIS-TOD-SEL
    Chassis scoped: LC_TX_SELECT
    Router scoped : None
  Uses frequency selection
  Used for local line interface output
  S  Input                     Last Selection Point         QL  Pri  Status
  == ========================  ========================  =====  ===  ===========
  1  Internal0 [0/RP0/CPU0]    n/a                         SEC  255  Holdover
     CPRIoE0/0/0/7             0/RP0/CPU0 ETH_RXMUX 3      PRC  100  Available
Selection point: 1588-SEL (2 inputs, 1 selected)
  Last programmed 6d00h ago, and selection made 02:54:01 ago
  Next selection points
    SPA scoped    : None
    Node scoped   : None
    Chassis scoped: None
    Router scoped : None
  Uses frequency selection
  S  Input                     Last Selection Point         QL  Pri  Status
  == ========================  ========================  =====  ===  ===========
  1  Internal0 [0/RP0/CPU0]    n/a                         SEC  255  Holdover
     CPRIoE0/0/0/7             0/RP0/CPU0 ETH_RXMUX 3      PRC  100  Available
Selection point: CHASSIS-TOD-SEL (1 inputs, 1 selected)
  Last programmed 02:54:12 ago, and selection made 02:54:12 ago
  Next selection points
    SPA scoped    : None
    Node scoped   : None
    Chassis scoped: None
    Router scoped : None
  Uses time-of-day selection
  S  Input                     Last Selection Point      Pri  Time  Status
  == ========================  ========================  ===  ====  ===========
  1  Internal0 [0/RP0/CPU0]    0/RP0/CPU0 T0-SEL-B 1     255  No    Available
Selection point: ETH_RXMUX (1 inputs, 1 selected)
  Last programmed 6d00h ago, and selection made 6d00h ago
  Next selection points
    SPA scoped    : None
    Node scoped   : T0-SEL-B 1588-SEL
    Chassis scoped: None
    Router scoped : None
  Uses frequency selection
  S  Input                     Last Selection Point         QL  Pri  Status
  == ========================  ========================  =====  ===  ===========
  3  CPRIoE0/0/0/7             n/a                         PRC  100  Available



RP/0/RP0/CPU0:R3# run
Mon Jan 20 17:20:01.237 UTC
[node0_RP0_CPU0:~]$
[node0_RP0_CPU0:~]$tmgctrl_client
tmgctrl>pll_rd 0x210
Read PLL reg 0x33f210 = 0x03
tmgctrl>pll_rd 0x120
Read PLL reg 0x33f120 = 0xf1
tmgctrl>
tmgctrl>exit
[node0_RP0_CPU0:~]$exit
logout
RP/0/RP0/CPU0:JAG1#
RP/0/RP0/CPU0:JAG1#show frequency synchronization interfaces
Mon Jan 20 17:21:41.994 UTC
Interface CPRIoE0/0/0/7 (up)
  Assigned as input for selection
  Wait-to-restore time 0 minutes
  SSM Disabled
  Input:
    Up
    Configured QL: Opt-I/PRC
    Effective QL: Opt-I/PRC, Priority: 100, Time-of-day Priority 100
    Supports frequency
  Output:
    Selected source: Internal0 [0/RP0/CPU0]
    Selected source QL: Opt-I/SEC
  Next selection points: ETH_RXMUX

On R4:

RP/0/RP0/CPU0:R4# show frequency synchronization selection
Mon Jan 20 17:22:55.884 UTC
Node 0/RP0/CPU0:
==============
Selection point: T0-SEL-B (3 inputs, 1 selected)
  Last programmed 5d23h ago, and selection made 03:01:38 ago
  Next selection points
    SPA scoped    : None
    Node scoped   : CHASSIS-TOD-SEL
    Chassis scoped: LC_TX_SELECT
    Router scoped : None
  Uses frequency selection
  Used for local line interface output
  S  Input                     Last Selection Point         QL  Pri  Status
  == ========================  ========================  =====  ===  ===========
  1  TenGigE0/0/0/14           0/RP0/CPU0 ETH_RXMUX 1      PRC  100  Locked
     TenGigE0/0/0/16           0/RP0/CPU0 ETH_RXMUX 2      PRC  100  Available
     Internal0 [0/RP0/CPU0]    n/a                         SEC  255  Available
Selection point: 1588-SEL (3 inputs, 1 selected)
  Last programmed 5d23h ago, and selection made 03:01:38 ago
  Next selection points
    SPA scoped    : None
    Node scoped   : None
    Chassis scoped: None
    Router scoped : None
  Uses frequency selection
  S  Input                     Last Selection Point         QL  Pri  Status
  == ========================  ========================  =====  ===  ===========
  1  TenGigE0/0/0/14           0/RP0/CPU0 ETH_RXMUX 1      PRC  100  Locked
     TenGigE0/0/0/16           0/RP0/CPU0 ETH_RXMUX 2      PRC  100  Available
     Internal0 [0/RP0/CPU0]    n/a                         SEC  255  Available
Selection point: CHASSIS-TOD-SEL (1 inputs, 1 selected)
  Last programmed 6d01h ago, and selection made 6d01h ago
  Next selection points
    SPA scoped    : None
    Node scoped   : None
    Chassis scoped: None
    Router scoped : None
  Uses time-of-day selection
  S  Input                     Last Selection Point      Pri  Time  Status
  == ========================  ========================  ===  ====  ===========
  1  TenGigE0/0/0/14           0/RP0/CPU0 T0-SEL-B 1     100  No    Available

Selection point: ETH_RXMUX (2 inputs, 2 selected)
  Last programmed 5d23h ago, and selection made 5d23h ago
  Next selection points
    SPA scoped    : None
    Node scoped   : T0-SEL-B 1588-SEL
    Chassis scoped: None
    Router scoped : None
  Uses frequency selection
  S  Input                     Last Selection Point         QL  Pri  Status
  == ========================  ========================  =====  ===  ===========
  1  TenGigE0/0/0/14           n/a                         PRC  100  Available
  2  TenGigE0/0/0/16           n/a                         PRC  100  Available

RP/0/RP0/CPU0:R4# show frequency synchronization interfaces
Mon Jan 20 17:23:29.828 UTC
Interface TenGigE0/0/0/14 (up)
  Assigned as input for selection
  Wait-to-restore time 0 minutes
  SSM Disabled
  Input:
    Up
    Configured QL: Opt-I/PRC
    Effective QL: Opt-I/PRC, Priority: 100, Time-of-day Priority 100
    Supports frequency
  Output:
    Selected source: TenGigE0/0/0/14
    Selected source QL: Opt-I/PRC
    Output is squelched
  Next selection points: ETH_RXMUX
Interface TenGigE0/0/0/16 (up)
  Assigned as input for selection
  Wait-to-restore time 0 minutes
  SSM Disabled
  Input:
    Up
    Configured QL: Opt-I/PRC
    Effective QL: Opt-I/PRC, Priority: 100, Time-of-day Priority 100
    Supports frequency
  Output:
    Selected source: TenGigE0/0/0/14
    Selected source QL: Opt-I/PRC
  Next selection points: ETH_RXMUX