PfRv3 Transit Site Support

Starting with Cisco IOS XE Release 3.15S and Cisco IOS Release 15.5(2)T release, Performance Routing version 3 (PfRv3) supports multiple data centers at the hub site. The multi-data center or the transit site support feature enables service providers to scale their network infrastructure, and load-balance the traffic when required.

Feature Information for PfRv3 Transit Site Support

The following table provides release information about the feature or features described in this module. This table lists only the software release that introduced support for a given feature in a given software release train. Unless noted otherwise, subsequent releases of that software release train also support that feature.

Use Cisco Feature Navigator to find information about platform support and Cisco software image support. To access Cisco Feature Navigator, go to www.cisco.com/go/cfn. An account on Cisco.com is not required.
Table 1. Feature Information for PfRv3 Transit Site Support

Feature Name

Releases

Feature Information

PfRv3 Transit Site Support

15.5(2)T

Cisco IOS XE Release 3.15S

The PfRv3 Transit Site Support feature enables service providers to configure multiple-data centers at the hub site.

The following commands were modified by this feature: master (domain VRF configuration), domain (interface configuration) .

Prerequisites for PfRv3 Transit Site Support

  • Upgrade all branch sites, hub, and transit sites with latest Cisco IOS image to enable transit site support feature.

Restrictions for PfRv3 Transit Site Support

  • Multiple next hops are supported only on hub or transit hub.

  • Basic tunnel function is not supported between an old Cisco IOS release version and a new version, if transit site support is enabled.

  • Hub sites must be connected by a Layer 3 routed link, which provides primary routing between the hub sites. Routing between hub sites over the DMVPN network is not supported

Information About PfRv3 Transit Site Support

Information About Transit Site Support

The multi-data center or the transit site support feature enables service providers to scale their network infrastructure, and load-balance the traffic when required. The multi-data center support enables all the hub sites to be connected with all the branch sites in an enterprise network. For example, in a use case scenario, an organization with two data centers and a single branch site, the branch site can communicate with the master-hub controller through the two next-hops (hub-branch routers) located at the hub site. If one hub-border router is down, then the branch site can still communicate through the second hub-border router. To differentiate the traffic from different hub-border routers, a path-id is configured on each interface of every channel. The branch router determines the inbound traffic based on the path-id of hub-branch routers. A path-id is a unique 32-bit number for a path between two sites.

PfRv3 Transit Site Use Case Scenarios

The transit site support feature supports the following use case scenarios:

  • Single data center with multiple borders

  • Dual data center with multiple borders

  • Dual data center with same prefix

Single Data Center with Multiple Borders

In the following illustration, spoke A (R10) is connected to two (BR1 and BR2) DMVPN hubs in a single Dynamic Multipoint VPN (DMVPN) domain. There are two paths and two next-hops to the hub site from the spoke A. To differentiate traffic from different ISP paths, a path-id is added on each domain path. Use the domain domain-name path path-name path-id command to configure the path-ids.

Figure 1. Single Data Center with Multiple Borders

Dual Data Center with Multiple Borders

In the following illustration, the two data centers are connected to all the branch sites. You can use both the data centers in active mode and use separate prefixes for both the data centers. To differentiate the traffic originating from different data centers, a transit-id is assigned to each data center. The valid range for a transit-id is from 1 to 62. By default, 0 is assigned to the master hub. Use the master transit command to configure the transit-id.

Figure 2. Dual Data Center with Multiple Borders

Dual Data Center with Same Prefix

In the following illustration, two data centers are connected to all the branch sites. However, in this scenario both the data centers are active and load-balance the traffic. If one data center is down, then traffic is routed through the other data center. Both the data centers share the same prefix.

Figure 3. Dual Data Center with Same Prefix

How to Configure Transit Site Support

Configuring Transit Hub

Before you begin

Configure the primary hub before configuring the transit hub.


Note

In the current release, transit hub support is available only on Cisco ASR 1000 Series Aggregation Services Routers and Cisco 4000 Series Integrated Services Routers.


Note

All policies are configured on the primary hub-master controller.

SUMMARY STEPS

  1. enable
  2. configure terminal
  3. interface loopback interface-number
  4. exit
  5. domain {domain-name | default }
  6. vrf {vrf-name | default }
  7. master transit pop-id
  8. source-interface loopback interface-number
  9. site-prefixes prefix-list site -list
  10. hub ip-address
  11. exit
  12. end
  13. (Optional) show domain domain-name master status

DETAILED STEPS

  Command or Action Purpose
Step 1

enable

Example:


Device> enable

Enables privileged EXEC mode.

  • Enter your password if prompted.

Step 2

configure terminal

Example:


Device# configure terminal

Enters global configuration mode.

Step 3

interface loopback interface-number

Example:


Device(config)# interface Loopback0

Enters interface configuration mode.

Step 4

exit

Example:


Device(config-if)# exit

Exits interface configuration mode and returns to global configuration mode.

Step 5

domain {domain-name | default }

Example:

Device(config)# domain default

Enters domain configuration mode.

Note 

You can either configure a default domain or define a specific domain for the transit hub configuration. If you are defining a specific domain, for example "domain-cisco", you must configure the same domain for all devices for PfRv3 configuration.

Step 6

vrf {vrf-name | default }

Example:

Device(config-domain)# vrf default

Configures default Virtual Routing and Forwarding (VRF) instances for the default or specific domain.

Step 7

master transit pop-id

Example:

Device(config-domain-vrf)# master transit 1

Enters master-controller configuration mode and configures the master as a transit hub. The valid range for a pop-id is from 1 to 62.

Step 8

source-interface loopback interface-number

Example:

Device(config-domain-vrf-mc)# source-interface Loopback0

Configures the loopback used as a source for peering with other sites or master controller.

Note 

The source-interface loopback also serves as a site ID of a particular site (hub or branch) on the master controller.

Step 9

site-prefixes prefix-list site -list

Example:

Device(config-domain-vrf-mc)# site-prefixes prefix-list Data_Center_1

Configures the prefix-list containing list of site prefixes.

Note 

You must configure the static-site prefix list for a hub and transit sites.

Step 10

hub ip-address

Example:

Device(config-domain-vrf-mc)# hub 10.8.3.3

Configures the hub for the transit site.

Step 11

exit

Example:

Device(config-domain-vrf-mc)# exit

Exits from master controller configuration mode and returns to domain configuration mode.

Note 

Exit from domain configuration mode and enter in global configuration mode using the exit command.

Step 12

end

Example:

Device(config)# end

Exits configuration mode and returns to privileged EXEC mode.

Step 13

(Optional) show domain domain-name master status

Example:

Device# show domain one master status
 (Optional)

Use this show command to display the status of a master controller.

Configuring Transit Site Border Routers


Note

In Cisco IOS XE Release 3.15S and Cisco IOS Release 15.5(2)T release, the transit site support is available only on Cisco ASR 1000 Series Aggregation Services Routers and Cisco 4000 Series Integrated Services Routers.

In a transit site support scenario, you must configure hub-border routers with the following:

  • The source interface of the border router

  • The IP address of the hub-master controller

  • The domain path name on external interfaces

  • The domain path ID for each external interface

To configure multiple hub-border routers to the same ISP path, perform the following task on each hub-border router.

SUMMARY STEPS

  1. enable
  2. configure terminal
  3. interface loopback interface-number
  4. ip address ip-address-mask
  5. exit
  6. domain {domain-name | default }
  7. vrf {vrf-name | default }
  8. border
  9. source-interface loopback interface-number
  10. master ip-address
  11. exit
  12. exit
  13. exit
  14. interface tunnel-name
  15. ip address ip-address mask
  16. description description-line
  17. domain domain-name path path-name path-id path-id
  18. end
  19. (Optional) show domain domain-name border status

DETAILED STEPS

  Command or Action Purpose
Step 1

enable

Example:


Device> enable

Enables privileged EXEC mode.

  • Enter your password if prompted.

Step 2

configure terminal

Example:


Device# configure terminal

Enters global configuration mode.

Step 3

interface loopback interface-number

Example:


Device(config)# interface Loopback0

Enters interface configuration mode.

Step 4

ip address ip-address-mask

Example:


Device(config-if)# ip address 10.9.4.4 255.255.255.255

Configures an IP address for an interface on the hub-border router (Border Router 1).

Step 5

exit

Example:


Device(config-if)# exit

Exits interface configuration mode and returns to global configuration mode.

Step 6

domain {domain-name | default }

Example:

Device(config)# domain default

Enters domain configuration mode.

Step 7

vrf {vrf-name | default }

Example:

Device(config-domain)# vrf default

Configures Virtual Routing and Forwarding (VRF) for the default domain.

Note 

You can configure specific VRF definition for the hub-border configuration.

Step 8

border

Example:

Device(config-domain-vrf)# border

Enters border configuration mode and configures the device as border.

Step 9

source-interface loopback interface-number

Example:

Device(config-domain-vrf-br)# source-interface Loopback0

Configures the loopback used as a source for peering with other sites or master controller.

Step 10

master ip-address

Example:

Device(config-domain-vrf-br)# master 10.9.3.3

Configures the IP address of the hub-master controller.

Step 11

exit

Example:

Device(config-domain-vrf-br)# exit

Exits border configuration mode and enters VRF configuration mode.

Step 12

exit

Example:

Device(config-domain-vrf)# exit

Exits VRF configuration mode and enters domain configuration mode.

Step 13

exit

Example:

Device(config-domain)# exit

Exits domain configuration mode and enters global configuration mode.

Step 14

interface tunnel-name

Example:

Device(config)# interface Tunnel100

Enters interface configuration mode.

Step 15

ip address ip-address mask

Example:

Device(config-if)# ip address 10.0.100.84 255.255.255.0

Configures an IP address for the tunnel interface.

Step 16

description description-line

Example:

Device1(config-if)# description primary path
Device2(config-if)# description secondary path

Configures a description to associate with an ISP path.

Step 17

domain domain-name path path-name path-id path-id

Example:

Device(config-if)# domain default path MPLS path-id 1

Configures the Internet Service Provider (ISP) associated with the domain and the path. There are two types of external interfaces, enterprise link such as DMVPN tunnel interface and internet -bound interface. Multiple next hop is supported only on DMVPN tunnel interfaces. The path-id is a unique identifier for each path in a domain. Valid values for a path-id are from 1 to 62.

We recommend using front VRF on the tunnel interface for enterprise links.

Note 

You can configure multiple ISPs. If you are defining specific domain name for example, domain_cisco, you must specify the same domain name for configuring ISP paths.

You must assign a unique path-id for all the paths that are connected from hub-border routers to the same ISP domain.

Step 18

end

Example:

Device(config-if)# end

Exits interface configuration mode and returns to privileged EXEC mode.

Step 19

(Optional) show domain domain-name border status

Example:

Device# show domain default border status
 (Optional)

Use this show command to display the status of a border router.

What to do next

Verifying PfRv3 Transit Site Support

Verifying PfRv3 Transit Site Support

The show commands can be entered in any order.

Before you begin

Configure multiple DMVPN paths from hub-border routers or from transit-hub border routers.

SUMMARY STEPS

  1. show domain domain-name master channels
  2. show domain domain-name border channel
  3. show domain domain-name master site-prefix
  4. show domain domain-name border site-prefix
  5. show domain domain-name master channels dst-site-id destination-site-id

DETAILED STEPS

Step 1

show domain domain-name master channels

Displays channel information of the hub-master controller.

Example:

HubMC# show domain default master channels

---------------------------------------------------------------------------------------------
Channel Id: 8  Dst Site-Id: 10.2.11.11  Link Name: MPLS  DSCP: default [0] pfr-label: 0:0 | 2:30 [0x21E] TCs: 0
  Channel Created: 03:19:14 ago
  Provisional State: Initiated and open
  Operational state: Available but unreachable
  Channel to hub: FALSE
  Interface Id: 11
  Supports Zero-SLA: Yes
  Muted by Zero-SLA: No
  Estimated Channel Egress Bandwidth: 0 Kbps
  Immitigable Events Summary:
   Total Performance Count: 0, Total BW Count: 0
  ODE Stats Bucket Number: 1
   Last Updated  : 00:00:21 ago
    Packet Count  : 0
    Byte Count    : 0
    One Way Delay  : N/A
    Loss Rate Pkts : N/A
    Loss Rate Bytes: N/A
    Jitter Mean    : N/A
    Unreachable   : TRUE
  ODE Stats Bucket Number: 2
   Last Updated  : 00:00:52 ago
    Packet Count  : 0
    Byte Count    : 0
    One Way Delay  : N/A
    Loss Rate Pkts : N/A
    Loss Rate Bytes: N/A
    Jitter Mean    : N/A
    Unreachable   : TRUE
   TCA Statistics:
      Received:355 ; Processed:354 ; Unreach_rcvd:355
  Latest TCA Bucket
   Last Updated  : 00:00:21 ago
    Local unreachable TCA received(Check for stale TCA 00:00:09 later)
.
.
.
---------------------------------------------------------------------------------------------
Step 2

show domain domain-name border channel

Displays the information of border router channels at the hub site.

Example:

HubBR# show domain default border channels

---------------------------------------------------------------------------------------------
Border Smart Probe Stats:

 Smart probe parameters:
   Source address used in the Probe: 10.2.10.10
   Unreach time: 1000 ms
   Probe source port: 18000
   Probe destination port: 19000
   Interface Discovery: ON
   Probe freq for channels with traffic :10 secs
   Discovery Probes: OFF
   Number of transit probes consumed :29
   Number of transit probes re-routed: 0
   DSCP's using this: [0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37] [38] [39] [40] [41] [42] [43] [44] [45] [46] [47] [48] [49] [50] [51] [52] [53] [54] [55] [56] [57] [58] [59] [60] [61] [62] [63] [64]
   All the other DSCPs use the default interval: 10 secs

Channel id: 20
  Channel create time: 06:42:54 ago
  Site id : 10.2.11.11
  DSCP : default[0]
  Service provider : MPLS
  Pfr-Label : 0:0 | 0:0 [0x0]
  exit path-id: 0
  Exit path-id sent on wire: 0
  Number of Probes sent : 77407
  Number of Probes received : 75949
  Last Probe sent : 00:00:00 ago
  Last Probe received : 00:00:00 ago
  Channel state : Initiated and open
  Channel next_hop : 10.0.100.11
  RX Reachability : Reachable
  TX Reachability : Reachable
  Channel is sampling 0 flows
  Channel remote end point: 10.0.100.11
  Channel to hub: FALSE
  Version: 3
  Supports Zero-SLA: Yes
  Muted by Zero-SLA: No
  Probe freq with traffic : 1 in 10000 ms

.
.
.
---------------------------------------------------------------------------------------------
Step 3

show domain domain-name master site-prefix

Displays the details of site-prefixes configured to the master hub.

Example:

HubMC# show domain default master site-prefix

---------------------------------------------------------------------------------------------
Load for five secs: 0%/0%; one minute: 0%; five minutes: 0%
Time source is NTP, 11:28:29.421 CET Tue Mar 17 2015

  Change will be published between 5-60 seconds
  Next Publish 00:33:03 later
  Prefix DB Origin: 10.9.3.3
  Prefix Flag: S-From SAF; L-Learned; T-Top Level; C-Configured; M-shared

Site-id              Site-prefix          Last Updated         DC Bitmap  Flag
--------------------------------------------------------------------------------
10.2.10.10            10.1.10.0/24         01:25:15 ago         0x0         S
10.2.11.11            10.1.11.0/24         01:25:19 ago         0x0         S
10.2.10.10            10.2.10.10/32        01:25:15 ago         0x0         S
10.2.11.11            10.2.11.11/32        01:25:19 ago         0x0         S
10.2.12.12            10.2.12.12/32        01:28:54 ago         0x0         S
10.8.3.3              10.8.3.3/32          01:28:47 ago         0x1         S
10.9.3.3              10.8.0.0/16          01:28:47 ago         0x5         C,M
10.8.3.3              10.8.0.0/16          01:28:47 ago         0x5         C,M
10.9.3.3              10.9.3.3/32          03:29:04 ago         0x4         L
10.9.3.3              10.9.0.0/16          01:28:47 ago         0x5         C,M
10.8.3.3              10.9.0.0/16          01:28:47 ago         0x5         C,M
255.255.255.255      *10.0.0.0/8           01:28:47 ago         0x1         S,T

---------------------------------------------------------------------------------------------
Step 4

show domain domain-name border site-prefix

Displays the details of site-prefixes configured on the border.

Example:

HubBR# show domain default border site-prefix

---------------------------------------------------------------------------------------------

Prefix Flag: S-From SAF; L-Learned; T-Top Level; C-Configured; M-shared

Site-id              Site-prefix          Last Updated         DC Bitmap  Flag
--------------------------------------------------------------------------------
10.2.10.10            10.1.10.0/24         00:36:58 ago         0x0         S
10.2.11.11            10.1.11.0/24         00:37:02 ago         0x0         S
10.2.10.10            10.2.10.10/32        00:36:58 ago         0x0         S
10.2.11.11            10.2.11.11/32        00:37:02 ago         0x0         S
10.2.12.12            10.2.12.12/32        00:40:37 ago         0x0         S
10.8.3.3              10.8.3.3/32          00:40:29 ago         0x1         S
10.9.3.3              10.8.0.0/16          00:38:40 ago         0x5         S,C,M
10.8.3.3              10.8.0.0/16          00:38:40 ago         0x5         S,C,M
10.9.3.3              10.9.3.3/32          00:38:40 ago         0x4         S
10.9.3.3              10.9.0.0/16          00:38:40 ago         0x5         S,C,M
10.8.3.3              10.9.0.0/16          00:38:40 ago         0x5         S,C,M
255.255.255.255      *10.0.0.0/8           00:40:29 ago         0x1         S,T
---------------------------------------------------------------------------------------------
Step 5

show domain domain-name master channels dst-site-id destination-site-id

Displays the details of destination site-ids configured with hub-master controller.

Note 

Use this command on a spoke or a branch device to view the details of the destination site-ids.

Example:

BR# show domain default master channels dst-site-id 10.8.3.3   

---------------------------------------------------------------------------------------------

Legend: * (Value obtained from Network delay:)

Channel Id: 27  Dst Site-Id: 10.8.3.3  Link Name: INET  DSCP: default [0] pfr-label: 0:20 | 0:0 [0x140000] TCs: 0
  Channel Created: 01:16:34 ago
  Provisional State: Initiated and open
  Operational state: Available
  Channel to hub: TRUE
  Interface Id: 12
  Supports Zero-SLA: Yes
  Muted by Zero-SLA: No
  Estimated Channel Egress Bandwidth: 5 Kbps
  Immitigable Events Summary:
   Total Performance Count: 0, Total BW Count: 0
  Site Prefix List
    10.8.3.3/32 (Active)
    10.8.0.0/16 (Active)
    10.9.0.0/16 (Standby)
  ODE Stats Bucket Number: 1
   Last Updated  : 00:00:24 ago
    Packet Count  : 562
Byte Count    : 47208
    One Way Delay : 71 msec*
    Loss Rate Pkts: 0.0 %
    Loss Rate Byte: 0.0 %
    Jitter Mean   : 619 usec
    Unreachable   : FALSE
  ODE Stats Bucket Number: 2
   Last Updated  : 00:00:54 ago
    Packet Count  : 558
    Byte Count    : 46872
    One Way Delay : 55 msec*
    Loss Rate Pkts: 0.0 %
    Loss Rate Byte: 0.0 %
    Jitter Mean   : 556 usec
    Unreachable   : FALSE
   TCA Statistics:
      Received:133 ; Processed:133 ; Unreach_rcvd:0
  Latest TCA Bucket
   Last Updated  : 00:00:24 ago
    One Way Delay : 71 msec*
    Loss Rate Pkts: NA
    Loss Rate Byte: NA
    Jitter Mean   : NA
    Unreachability: FALSE
.
.
.
---------------------------------------------------------------------------------------------

Configuration Examples for PfRv3 Transit Site Support

Example: Configuring Transit Site Support

Example: Configuring Data Center 1 (DC1) Devices

In this use case scenario, an enterprise organization has two data centers with multiple-border routers connected to the same ISP domain. The branch-border routers can reach the hub-master controller through multiple next-hops.

Figure 4. PfRv3 Transit Hub Topology
In this example, the following routers are used:

  • Hub Master Controller — Cisco ASR 1002-X router configured with an embedded services processor (ESP) default bandwidth of 5 Gbps upgradable with software licensing options to 10 Gbps, 20 Gbps, and 36 Gbps.

  • Hub Border Routers — Cisco ASR 1000 Series Embedded Services Processor 2

  • Branch Routers — Cisco 4451X Integrated Services Router.

Configure the interfaces on master hub controller (R82) 

HubMC> enable
HubMC# configure terminal
HubMC(config)# interface Loopback0
HubMC(config-if)# ip address 10.8.3.3 255.255.255.255
HubMC(config-if)# exit

Configure the device as hub-master controller 

HubMC(config)# domain default
HubMC(config-domain)# vrf default
HubMC(config-domain-vrf)# master hub
HubMC(config-domain-vrf-mc)# source-interface Loopback0
HubMC(config-domain-vrf-mc)# enterprise-prefix prefix-list ENTERPRISE_PREFIX
HubMC(config-domain-vrf-mc)# site-prefixes prefix-list DC1_PREFIX
HubMC(config-domain-vrf-mc)# exit

Configure IP prefix-lists 

HubMC(config)# ip prefix-list DC1_PREFIX seq 10 permit 10.8.0.0/16
HubMC(config)# ip prefix-list DC1_PREFIX seq 10 permit 10.9.0.0/16
HubMC(config)# ip prefix-list ENTERPRISE_PREFIX seq 10 permit 10.0.0.0/8

Configure domain policies on hub master controller 

HubMC(config)# domain default
HubMC(config-domain)# vrf default
HubMC(config-domain-vrf)# master hub
HubMC(config-domain-vrf-mc)# source-interface Loopback0
HubMC(config-domain-vrf-mc)# site-prefixes prefix-list DC1_PREFIX
HubMC(config-domain-vrf-mc)# load-balance
HubMC(config-domain-vrf-mc)# enterprise-prefix  prefix-list ENTERPRISE_PREFIX

HubMC(config-domain-vrf-mc)# class VOICE sequence 10
HubMC(config-domain-vrf-mc-class)# match dscp ef policy custom
HubMC(config-domain-vrf-mc-class-type)# priority 2 loss threshold 5
HubMC(config-domain-vrf-mc-class-type)# priority 1 one-way-delay  threshold 150
HubMC(config-domain-vrf-mc-class-type)# exit
HubMC(config-domain-vrf-mc-class)# path-preference MPLS fallback INET
HubMC(config-domain-vrf-mc-class)# exit

HubMC(config-domain-vrf-mc)# class VIDEO sequence 20
HubMC(config-domain-vrf-mc-class)# match dscp af41 policy custom
HubMC(config-domain-vrf-mc-class-type)# priority 2 loss threshold 5
HubMC(config-domain-vrf-mc-class-type)# priority 1 one-way-delay  threshold 150
HubMC(config-domain-vrf-mc-class-type)# exit
HubMC(config-domain-vrf-mc-class)# match dscp cs4 policy custom
HubMC(config-domain-vrf-mc-class-type)# priority 2 loss threshold 5
HubMC(config-domain-vrf-mc-class-type)# priority 1 one-way-delay  threshold 150
HubMC(config-domain-vrf-mc-class-type)# exit
HubMC(config-domain-vrf-mc-class)# path-preference INET fallback MPLS
HubMC(config-domain-vrf-mc-class)# exit


HubMC(config-domain-vrf-mc)# class CRITICAL sequence 30
HubMC(config-domain-vrf-mc-class)# match dscp af31 policy custom
HubMC(config-domain-vrf-mc-class-type)# priority 2 loss threshold 10
HubMC(config-domain-vrf-mc-class-type)# priority 1 one-way-delay threshold 600
HubMC(config-domain-vrf-mc-class-type)# exit
HubMC(config-domain-vrf-mc-class)# path-preference MPLS fallback INET
HubMC(config-domain-vrf-mc)# class DEFAULT sequence 100
HubMC(config-domain-vrf-mc-class)# match dscp default policy custom
HubMC(config-domain-vrf-mc-class-type)# priority 2 loss threshold 5
HubMC(config-domain-vrf-mc-class-type)# priority 1 one-way-delay threshold 50
HubMC(config-domain-vrf-mc-class-type)# priority 3 jitter threshold 200000
HubMC(config-domain-vrf-mc-class-type)# exit

Configure hub border routers on DC1 (R84) 

BR84> enable
BR84# configure terminal
BR84(config)# interface Loopback0
BR84(config-if)# ip address 10.8.4.4 255.255.255.255
BR84(config-if)exit

Configure the device as border router (BR84) 

BR84(config)# domain default
BR84(config-domain)# vrf default
BR84(config-domain-vrf)# border
BR84(config-domain-vrf-br)# source-interface Loopback0
BR84(config-domain-vrf-br)# master 10.8.3.3
BR84(config-domain-vrf-br)# exit

Configure tunnel from BR84 to DMVPN1 (MPLS)Link 

BR84(config)# interface Tunnel100
BR84(config-if)# bandwidth 100000
BR84(config-if)# ip address 10.0.100.84 255.255.255.0
BR84(config-if)# no ip redirects
BR84(config-if)# ip mtu 1400
BR84(config-if)# ip nhrp authentication cisco
BR84(config-if)# ip nhrp map multicast dynamic
BR84(config-if)# ip nhrp network-id 1
BR84(config-if)# ip nhrp holdtime 60
BR84(config-if)# ip nhrp redirect
BR84(config-if)# ip tcp adjust-mss 1360
BR84(config-if)# load-interval 30
BR84(config-if)# delay 1000
BR84(config-if)# tunnel source Ethernet0/1
BR84(config-if)# tunnel mode gre multipoint
BR84(config-if)# tunnel key 100
BR84(config-if)# tunnel vrf IWAN-TRANSPORT-1
BR84(config-if)# domain path MPLS path-id 10

Configure hub border routers on DC1 (R85) 

BR85> enable
BR85# configure terminal
BR85(config)# interface Loopback0
BR85(config-if)# ip address 10.8.5.5 255.255.255.255
BR85(config-if)exit

Configure the device as border router (BR85) 

BR85(config)# domain default
BR85(config-domain)# vrf default
BR85(config-domain-vrf)# border
BR85(config-domain-vrf-br)# source-interface Loopback0
BR85(config-domain-vrf-br)# master 10.8.3.3
BR85(config-domain-vrf-br)# exit

Configure tunnel from BR84 to DMVPN2 (INET)Link 

BR85(config)# interface Tunnel200
BR85(config-if)# bandwidth 5000
BR85(config-if)# ip address 10.0.200.85 255.255.255.0
BR85(config-if)# no ip redirects
BR85(config-if)# ip mtu 1400
BR85(config-if)# ip nhrp authentication cisco
BR85(config-if)# ip nhrp map multicast dynamic
BR85(config-if)# ip nhrp network-id 2
BR85(config-if)# ip nhrp holdtime 60
BR85(config-if)# ip nhrp redirect
BR85(config-if)# ip tcp adjust-mss 1360
BR85(config-if)# load-interval 30
BR85(config-if)# delay 1000
BR85(config-if)# tunnel source Ethernet0/1
BR85(config-if)# tunnel mode gre multipoint
BR85(config-if)# tunnel key 200
BR85(config-if)# tunnel vrf IWAN-TRANSPORT-2
BR85(config-if)# domain path INET path-id 20
Configure the interfaces on master hub controller (R92) 
HubMC> enable
HubMC# configure terminal
HubMC(config)# interface Loopback0
HubMC(config-if)# ip address 10.9.3.3 255.255.255.255
HubMC(config-if)# exit
Configure the device as transit-hub master controller 
HubMC(config)# domain default
HubMC(config-domain)# vrf default
HubMC(config-domain-vrf)# master transit 2
HubMC(config-domain-vrf-mc)# source-interface Loopback0
HubMC(config-domain-vrf-mc)# site-prefixes prefix-list DC2_PREFIX
HubMC(config-domain-vrf-mc)# hub 10.8.3.3
HubMC(config-domain-vrf-mc)# exit
Configure IP prefix-lists 
HubMC(config)# ip prefix-list DC2_PREFIX seq 10 permit 10.9.0.0/16
HubMC(config)# ip prefix-list DC2_PREFIX seq 20 permit 10.8.0.0/16
HubMC(config)# ip prefix-list ENTERPRISE_PREFIX seq 10 permit 10.0.0.0/8
Configure hub border routers on DC2 (R94) 
BR94> enable
BR94# configure terminal
BR94(config)# interface Loopback0
BR94(config-if)# ip address 10.9.4.4 255.255.255.255
BR94(config-if)exit
Configure the device as border router (BR94) 
BR94(config)# domain default
BR94(config-domain)# vrf default
BR94(config-domain-vrf)# border
BR94(config-domain-vrf-br)# source-interface Loopback0
BR94(config-domain-vrf-br)# master 10.9.3.3
BR94(config-domain-vrf-br)# exit
Configure tunnel from BR94 to DMVPN1 (MPLS)Link 
BR94(config)# interface Tunnel100
BR94(config-if)# bandwidth 1000
BR94(config-if)# ip address 10.0.100.94 255.255.255.0
BR94(config-if)# no ip redirects
BR94(config-if)# ip mtu 1400
BR94(config-if)# ip nhrp authentication cisco
BR94(config-if)# ip nhrp map multicast dynamic
BR94(config-if)# ip nhrp network-id 1
BR94(config-if)# ip nhrp holdtime 60
BR94(config-if)# ip nhrp redirect
BR94(config-if)# ip tcp adjust-mss 1360
BR94(config-if)# load-interval 30
BR94(config-if)# delay 1000
BR94(config-if)# tunnel source Ethernet0/1
BR94(config-if)# tunnel mode gre multipoint
BR94(config-if)# tunnel key 100
BR94(config-if)# tunnel vrf IWAN-TRANSPORT-1
BR94(config-if)# domain path MPLS path-id 30
Configure hub border routers on DC2 (R95) 
BR95> enable
BR95# configure terminal
BR95(config)# interface Loopback0
BR95(config-if)# ip address 10.9.5.5 255.255.255.255
BR95(config-if)exit
Configure the device as border router (BR95) 
BR95(config)# domain default
BR95(config-domain)# vrf default
BR95(config-domain-vrf)# border
BR95(config-domain-vrf-br)# source-interface Loopback0
BR95(config-domain-vrf-br)# master 10.9.3.3
BR95(config-domain-vrf-br)# exit
Configure tunnel from BR95 to DMVPN2 (INET)Link 
BR95(config)# interface Tunnel200
BR95(config-if)# bandwidth 1000
BR95(config-if)# ip address 10.0.200.95 255.255.255.0
BR95(config-if)# no ip redirects
BR95(config-if)# ip mtu 1400
BR95(config-if)# ip nhrp authentication cisco
BR95(config-if)# ip nhrp map multicast dynamic
BR95(config-if)# ip nhrp network-id 2
BR95(config-if)# ip nhrp holdtime 60
BR95(config-if)# ip nhrp redirect
BR95(config-if)# ip tcp adjust-mss 1360
BR95(config-if)# load-interval 30
BR95(config-if)# delay 1000
BR95(config-if)# tunnel source Ethernet0/1
BR95(config-if)# tunnel mode gre multipoint
BR95(config-if)# tunnel key 200
BR95(config-if)# tunnel vrf IWAN-TRANSPORT-2
BR95(config-if)# domain path INET path-id 40

Example: Configuring Branch Routers

Configure the interfaces (R10) 
R10> enable
R10# configure terminal
R10(config)# interface Loopback0
R10(config-if)# ip address 10.2.10.10 255.255.255.255
R10(config-if)# exit
Configure the device as branch-master controller (R10) 
R10(config)# domain default
R10(config-domain)# vrf default
R10(config-domain-vrf)# border
R10(config-domain-vrf-br)# source-interface Loopback0
R10(config-domain-vrf-br)# master local
R10(config-domain-vrf-br)# exit
R10(config-domain-vrf)# master branch
R10(config-domain-vrf-mc)# source-interface Loopback0
R10(config-domain-vrf-mc)# hub 10.8.3.3
Configure the tunnel interface and tunnel path from R10 
R10(config)# interface Tunnel100
R10(config-if)# bandwidth 400
R10(config-if)# ip address 10.0.100.10 255.255.255.0
R10(config-if)# no ip redirects
R10(config-if)# ip mtu 1400
R10(config-if)# ip nhrp authentication cisco
R10(config-if)# ip nhrp map multicast dynamic
R10(config-if)# ip nhrp network-id 1
R10(config-if)# ip nhrp holdtime 60
R10(config-if)# ip nhrp nhs 10.0.100.84 nbma 172.16.84.4 multicast
R10(config-if)# ip nhrp nhs 10.0.100.94 nbma 172.16.94.4 multicast
R10(config-if)# ip nhrp registration no-unique
R10(config-if)# ip nhrp registration timeout 60
R10(config-if)# ip nhrp shortcut
R10(config-if)# ip nhrp redirect
R10(config-if)# ip tcp adjust-mss 1360
R10(config-if)# load-interval 30
R10(config-if)# delay 1000
R10(config-if)# no nhrp route-watch
R10(config-if)# if-state nhrp
R10(config-if)# tunnel source Ethernet0/1
R10(config-if)# tunnel mode gre multipoint
R10(config-if)# tunnel key 100
R10(config-if)# tunnel vrf IWAN-TRANSPORT-1

R10(config)# interface Tunnel200
R10(config-if)# bandwidth 5000
R10(config-if)# ip address 10.0.200.10 255.255.255.0
R10(config-if)# no ip redirects
R10(config-if)# ip mtu 1400
R10(config-if)# ip nhrp authentication cisco
R10(config-if)# ip nhrp map multicast dynamic
R10(config-if)# ip nhrp network-id 2
R10(config-if)# ip nhrp holdtime 600
R10(config-if)# ip nhrp nhs 10.0.200.85 nbma 172.16.85.5 multicast
R10(config-if)# ip nhrp nhs 10.0.200.95 nbma 172.16.95.5 multicast
R10(config-if)# ip nhrp registration no-unique
R10(config-if)# ip nhrp registration timeout 60
R10(config-if)# ip nhrp shortcut
R10(config-if)# ip nhrp redirect
R10(config-if)# ip tcp adjust-mss 1360
R10(config-if)# load-interval 30
R10(config-if)# delay 1000
R10(config-if)# no nhrp route-watch
R10(config-if)# if-state nhrp
R10(config-if)# tunnel source Ethernet0/2
R10(config-if)# tunnel mode gre multipoint
R10(config-if)# tunnel key 200
R10(config-if)# tunnel vrf IWAN-TRANSPORT-2
Configure the interfaces (R11) 
R11> enable
R11# configure terminal
R11(config)# interface Loopback0
R11(config-if)# ip address 10.2.11.11 255.255.255.255
R11(config-if)# exit
Configure the device as branch master controller (R11) 
R11(config)# domain default
R11(config-domain)# vrf default
R11(config-domain-vrf)# border
R11(config-domain-vrf-br)# source-interface Loopback0
R11(config-domain-vrf-br)# master local
R11(config-domain-vrf-br)# exit
R11(config-domain-vrf)# master branch
R11(config-domain-vrf-mc)# source-interface Loopback0
R11(config-domain-vrf-mc)# hub 10.8.3.3
Configure the tunnel interface and tunnel path from R11 
R11(config)# interface Tunnel100
R11(config-if)# bandwidth 2000
R11(config-if)# ip address 10.0.100.11 255.255.255.0
R11(config-if)# no ip redirects
R11(config-if)# ip mtu 1400
R11(config-if)# ip nhrp authentication cisco
R11(config-if)# ip nhrp map multicast dynamic
R11(config-if)# ip nhrp network-id 1
R11(config-if)# ip nhrp holdtime 60
R11(config-if)# ip nhrp nhs 10.0.100.84 nbma 172.16.84.4 multicast
R11(config-if)# ip nhrp nhs 10.0.100.94 nbma 172.16.94.4 multicast
R11(config-if)# ip nhrp registration no-unique
R11(config-if)# ip nhrp registration timeout 60
R11(config-if)# ip nhrp shortcut
R11(config-if)# ip nhrp redirect
R11(config-if)# ip tcp adjust-mss 1360
R11(config-if)# load-interval 30
R11(config-if)# delay 1000
R11(config-if)# no nhrp route-watch
R11(config-if)# if-state nhrp
R11(config-if)# tunnel source Ethernet0/1
R11(config-if)# tunnel mode gre multipoint
R11(config-if)# tunnel key 100
R11(config-if)# tunnel vrf IWAN-TRANSPORT-1

R11(config)# interface Tunnel200
R11(config-if)# bandwidth 5000
R11(config-if)# ip address 10.0.200.11 255.255.255.0
R11(config-if)# no ip redirects
R11(config-if)# ip mtu 1400
R11(config-if)# ip nhrp authentication cisco
R11(config-if)# ip nhrp map multicast dynamic
R11(config-if)# ip nhrp network-id 2
R11(config-if)# ip nhrp holdtime 600
R11(config-if)# ip nhrp nhs 10.0.200.85 nbma 172.16.85.5 multicast
R11(config-if)# ip nhrp nhs 10.0.200.95 nbma 172.16.95.5 multicast
R11(config-if)# ip nhrp registration no-unique
R11(config-if)# ip nhrp registration timeout 60
R11(config-if)# ip nhrp shortcut
R11(config-if)# ip nhrp redirect
R11(config-if)# ip tcp adjust-mss 1360
R11(config-if)# load-interval 30
R11(config-if)# delay 1000
R11(config-if)# no nhrp route-watch
R11(config-if)# if-state nhrp
R11(config-if)# tunnel source Ethernet0/2
R11(config-if)# tunnel mode gre multipoint
R11(config-if)# tunnel key 200
R11(config-if)# tunnel vrf IWAN-TRANSPORT-2
Configure the interfaces (R12) 
R12> enable
R12# configure terminal
R12(config)# interface Loopback0
R12(config-if)# ip address 10.2.12.12 255.255.255.255
R12(config-if)# exit
Configure the device as branch-master controller (R12) 
R12(config)# domain default
R12(config-domain)# vrf default
R12(config-domain-vrf)# border
R12(config-domain-vrf-br)# source-interface Loopback0
R12(config-domain-vrf-br)# master local
R12(config-domain-vrf-br)# exit
R12(config-domain-vrf)# master branch
R12(config-domain-vrf-mc)# source-interface Loopback0
R12(config-domain-vrf-mc)# hub 10.8.3.3
Configure the tunnel interface and tunnel path from R12 
R12(config)# interface Tunnel100
R12(config-if)# bandwidth 400
R12(config-if)# ip address 10.0.100.12 255.255.255.0
R12(config-if)# no ip redirects
R12(config-if)# ip mtu 1400
R12(config-if)# ip nhrp authentication cisco
R12(config-if)# ip nhrp map multicast dynamic
R12(config-if)# ip nhrp network-id 1
R12(config-if)# ip nhrp holdtime 600
R12(config-if)# ip nhrp nhs 10.0.100.84 nbma 172.16.84.4 multicast
R12(config-if)# ip nhrp nhs 10.0.100.94 nbma 172.16.94.4 multicast
R12(config-if)# ip nhrp registration no-unique
R12(config-if)# ip nhrp registration timeout 60
R12(config-if)# ip nhrp shortcut
R12(config-if)# ip tcp adjust-mss 1360
R12(config-if)# load-interval 30
R12(config-if)# delay 1000
R12(config-if)# no nhrp route-watch
R12(config-if)# if-state nhrp
R12(config-if)# tunnel source Ethernet0/1
R12(config-if)# tunnel mode gre multipoint
R12(config-if)# tunnel key 100
R12(config-if)# tunnel vrf IWAN-TRANSPORT-1
Configure the interfaces (R13) 
R13> enable
R13# configure terminal
R13(config)# interface Loopback0
R13(config-if)# ip address 10.2.13.13 255.255.255.255
R13(config-if)# exit
Configure the device as a border router with R12 as the master controller (R13) 
R13(config)# domain default
R13(config-domain)# vrf default
R13(config-domain-vrf)# border
R13(config-domain-vrf-br)# source-interface Loopback0
R13(config-domain-vrf-br)# master 10.2.12.12
R13(config-domain-vrf-br)# exit
Configure the tunnel interface and tunnel path from R13 
R13(config)# interface Tunnel200
R13(config-if)# bandwidth 400
R13(config-if)# ip address 10.0.200.13 255.255.255.0
R13(config-if)# no ip redirects
R13(config-if)# ip mtu 1400
R13(config-if)# ip nhrp authentication cisco
R13(config-if)# ip nhrp network-id 2
R13(config-if)# ip nhrp holdtime 600
R13(config-if)# ip nhrp nhs 10.0.200.85 nbma 172.16.85.5 multicast
R13(config-if)# ip nhrp nhs 10.0.100.95 nbma 172.16.95.5 multicast
R13(config-if)# ip nhrp registration no-unique
R13(config-if)# ip nhrp registration timeout 60
R13(config-if)# ip nhrp shortcut
R13(config-if)# ip tcp adjust-mss 1360
R13(config-if)# load-interval 30
R13(config-if)# delay 1000
R13(config-if)# if-state nhrp
R13(config-if)# tunnel source Ethernet0/2
R13(config-if)# tunnel mode gre multipoint
R13(config-if)# tunnel key 200
R13(config-if)# tunnel vrf IWAN-TRANSPORT-2

Verifying PfRv3 Transit Site Configuration

To verify the PfRv3 transit site configuration, use the following show commands in any order: 
HubMC2# show domain default master status

----------------------------------------------------------------------------------------------
*** Domain MC Status ***

 Master VRF: Global

  Instance Type:    Transit
  POP ID:      2
  Instance id:      0
  Operational status:  Up
  Configured status:  Up
  Loopback IP Address: 10.9.3.3
  Load Balancing:
   Operational Status: Up
   Max Calculated Utilization Variance: 0%
   Last load balance attempt: 03:07:30 ago
   Last Reason:  Variance less than 20%
   Total unbalanced bandwidth:
         External links: 0 Kbps  Internet links: 0 Kbps
  Route Control: Enabled
  Mitigation mode Aggressive: Disabled
  Policy threshold variance: 20
  Minimum Mask Length: 28
  Minimum Requirement: Met

  Borders:
    IP address: 10.9.5.5
    Version: 2
    Connection status: CONNECTED (Last Updated 03:25:38 ago )
    Interfaces configured: 
Name: Tunnel200 | type: external | Service Provider: INET path-id:40 | Status: UP | Zero-SLA: NO
          Number of default Channels: 0


    Tunnel if: Tunnel0

    IP address: 10.9.4.4
    Version: 2
    Connection status: CONNECTED (Last Updated 03:25:37 ago )
    Interfaces configured:
      Name: Tunnel100 | type: external | Service Provider: MPLS path-id:30 | Status: DOWN
    Tunnel if: Tunnel0

---------------------------------------------------------------------------------------------               

HubMC2# show domain default master channels

---------------------------------------------------------------------------------------------
Channel Id: 8  Dst Site-Id: 10.2.11.11  Link Name: MPLS  DSCP: default [0] pfr-label: 0:0 | 2:30 [0x21E] TCs: 0
  Channel Created: 03:19:14 ago
  Provisional State: Initiated and open
  Operational state: Available but unreachable
  Channel to hub: FALSE
  Interface Id: 11
  Supports Zero-SLA: Yes
  Muted by Zero-SLA: No
  Estimated Channel Egress Bandwidth: 0 Kbps
  Immitigable Events Summary:
   Total Performance Count: 0, Total BW Count: 0
  ODE Stats Bucket Number: 1
   Last Updated  : 00:00:21 ago
    Packet Count  : 0
    Byte Count    : 0
    One Way Delay  : N/A
    Loss Rate Pkts : N/A
    Loss Rate Bytes: N/A
    Jitter Mean    : N/A
    Unreachable   : TRUE
  ODE Stats Bucket Number: 2
   Last Updated  : 00:00:52 ago
    Packet Count  : 0
    Byte Count    : 0
    One Way Delay  : N/A
    Loss Rate Pkts : N/A
    Loss Rate Bytes: N/A
    Jitter Mean    : N/A
    Unreachable   : TRUE
   TCA Statistics:
      Received:355 ; Processed:354 ; Unreach_rcvd:355
  Latest TCA Bucket
   Last Updated  : 00:00:21 ago
    Local unreachable TCA received(Check for stale TCA 00:00:09 later)
.
.
.
---------------------------------------------------------------------------------------------
HubMC2#  show domain default master site-capability device-capb path-id

----------------------------------------------------------------------------------------------

Site pop id : 1
Site mc type : Transit
Border Address : 10.9.4.4
Service provider: MPLS path-id: 30 if_index: 11
Border Address : 10.9.5.5
Service provider: INET path-id: 40 if_index: 11
 
---------------------------------------------------------------------------------------------


HubMC2# show domain default master site-prefix

---------------------------------------------------------------------------------------------
Load for five secs: 0%/0%; one minute: 0%; five minutes: 0%
Time source is NTP, 11:28:29.421 CET Tue Mar 17 2015

  Change will be published between 5-60 seconds
  Next Publish 00:33:03 later
  Prefix DB Origin: 10.9.3.3
  Prefix Flag: S-From SAF; L-Learned; T-Top Level; C-Configured; M-shared

Site-id              Site-prefix          Last Updated         DC Bitmap  Flag
--------------------------------------------------------------------------------
10.2.10.10            10.1.10.0/24         01:25:15 ago         0x0         S
10.2.11.11            10.1.11.0/24         01:25:19 ago         0x0         S
10.2.10.10            10.2.10.10/32        01:25:15 ago         0x0         S
10.2.11.11            10.2.11.11/32        01:25:19 ago         0x0         S
10.2.12.12            10.2.12.12/32        01:28:54 ago         0x0         S
10.8.3.3              10.8.3.3/32          01:28:47 ago         0x1         S
10.9.3.3              10.8.0.0/16          01:28:47 ago         0x5         C,M
10.8.3.3              10.8.0.0/16          01:28:47 ago         0x5         C,M
10.9.3.3              10.9.3.3/32          03:29:04 ago         0x4         L
10.9.3.3              10.9.0.0/16          01:28:47 ago         0x5         C,M
10.8.3.3              10.9.0.0/16          01:28:47 ago         0x5         C,M
255.255.255.255      *10.0.0.0/8           01:28:47 ago         0x1         S,T

---------------------------------------------------------------------------------------------
HubMC2# show domain default master policy

---------------------------------------------------------------------------------------------

Load for five secs: 0%/0%; one minute: 0%; five minutes: 0%
Time source is NTP, 11:31:10.977 CET Tue Mar 17 2015

  class VOICE sequence 10
    path-preference MPLS fallback INET
    class type: Dscp Based
      match dscp ef policy custom
        priority 2 packet-loss-rate threshold 5.0 percent
        priority 1 one-way-delay threshold 150 msec
        priority 2 byte-loss-rate threshold 5.0 percent

  class VIDEO sequence 20
    path-preference MPLS fallback INET
    class type: Dscp Based
      match dscp af41 policy custom
        priority 2 packet-loss-rate threshold 5.0 percent
        priority 1 one-way-delay threshold 150 msec
        priority 2 byte-loss-rate threshold 5.0 percent
      match dscp cs4 policy custom
        priority 2 packet-loss-rate threshold 5.0 percent
        priority 1 one-way-delay threshold 150 msec
        priority 2 byte-loss-rate threshold 5.0 percent

  class CRITICAL sequence 30
    path-preference MPLS fallback INET
    class type: Dscp Based
      match dscp af31 policy custom
        priority 2 packet-loss-rate threshold 10.0 percent
        priority 1 one-way-delay threshold 600 msec
        priority 2 byte-loss-rate threshold 10.0 percent
        Number of Traffic classes using this policy: 1

class DEFAULT0 sequence 100
    class type: Dscp Based
      match dscp default policy custom
        priority 2 packet-loss-rate threshold 5.0 percent
        priority 1 one-way-delay threshold 50 msec
        priority 3 jitter threshold 200000 usec
        priority 2 byte-loss-rate threshold 5.0 percent
        Number of Traffic classes using this policy: 1

  class default
      match dscp all

---------------------------------------------------------------------------------------------
HubMC2# show domain default master discovered   

---------------------------------------------------------------------------------------------
Load for five secs: 0%/0%; one minute: 0%; five minutes: 0%
Time source is NTP, 14:31:58.410 CET Tue Mar 17 2015


  *** Domain MC DISCOVERED sites ***

  Number of sites:  5
 *Traffic classes [Performance based][Load-balance based]

 Site ID: 255.255.255.255
  Site Discovered:06:32:33 ago
    Off-limits: Disabled
    DSCP :default[0]-Number of traffic classes[0][0]
    DSCP :af31[26]-Number of traffic classes[0][0]

 Site ID: 10.8.3.3
  Site Discovered:06:30:37 ago
    Off-limits: Disabled
    DSCP :default[0]-Number of traffic classes[0][0]
    DSCP :af31[26]-Number of traffic classes[0][0]

 Site ID: 10.2.10.10
  Site Discovered:06:30:37 ago
    Off-limits: Disabled
    DSCP :default[0]-Number of traffic classes[1][0]
    DSCP :af31[26]-Number of traffic classes[1][0]

Site ID: 10.2.11.11
  Site Discovered:06:30:34 ago
    Off-limits: Disabled
    DSCP :default[0]-Number of traffic classes[0][0]
    DSCP :af31[26]-Number of traffic classes[0][0]

 Site ID: 10.2.12.12
  Site Discovered:06:30:37 ago
    Off-limits: Disabled
    DSCP :default[0]-Number of traffic classes[0][0]
    DSCP :af31[26]-Number of traffic classes[0][0]
---------------------------------------------------------------------------------------------
BR94# show domain default border status   

---------------------------------------------------------------------------------------------

**** Border Status ****

Instance Status: UP
Present status last updated: 06:39:21 ago
Loopback: Configured Loopback0 UP (10.9.4.4)
Master: 10.9.3.3
Master version: 2
Connection Status with Master: UP
MC connection info: CONNECTION SUCCESSFUL
Connected for: 06:39:15
Route-Control: Enabled
Asymmetric Routing: Disabled
Minimum Mask length: 28
Sampling: off
Minimum Requirement: Met
External Wan interfaces:
     Name: Tunnel100 Interface Index: 11 SNMP Index: 8 SP: MPLS path-id: 30 Status: DOWN Zero-SLA: NO

Auto Tunnel information:

   Name:Tunnel0 if_index: 12
   Borders reachable via this tunnel:  10.9.5.5

---------------------------------------------------------------------------------------------
BR94# show domain default border site-prefix   

---------------------------------------------------------------------------------------------

Prefix Flag: S-From SAF; L-Learned; T-Top Level; C-Configured; M-shared

Site-id              Site-prefix          Last Updated         DC Bitmap  Flag
--------------------------------------------------------------------------------
10.2.10.10            10.1.10.0/24         00:36:58 ago         0x0         S
10.2.11.11            10.1.11.0/24         00:37:02 ago         0x0         S
10.2.10.10            10.2.10.10/32        00:36:58 ago         0x0         S
10.2.11.11            10.2.11.11/32        00:37:02 ago         0x0         S
10.2.12.12            10.2.12.12/32        00:40:37 ago         0x0         S
10.8.3.3              10.8.3.3/32          00:40:29 ago         0x1         S
10.9.3.3              10.8.0.0/16          00:38:40 ago         0x5         S,C,M
10.8.3.3              10.8.0.0/16          00:38:40 ago         0x5         S,C,M
10.9.3.3              10.9.3.3/32          00:38:40 ago         0x4         S
10.9.3.3              10.9.0.0/16          00:38:40 ago         0x5         S,C,M
10.8.3.3              10.9.0.0/16          00:38:40 ago         0x5         S,C,M
255.255.255.255      *10.0.0.0/8           00:40:29 ago         0x1         S,T
---------------------------------------------------------------------------------------------
R10# show domain default master channels dst-site-id 10.8.3.3   

---------------------------------------------------------------------------------------------

Legend: * (Value obtained from Network delay:)

Channel Id: 27  Dst Site-Id: 10.8.3.3  Link Name: INET  DSCP: default [0] pfr-label: 0:20 | 0:0 [0x140000] TCs: 0
  Channel Created: 01:16:34 ago
  Provisional State: Initiated and open
  Operational state: Available
  Channel to hub: TRUE
  Interface Id: 12
  Supports Zero-SLA: Yes
  Muted by Zero-SLA: No
  Estimated Channel Egress Bandwidth: 5 Kbps
  Immitigable Events Summary:
   Total Performance Count: 0, Total BW Count: 0
  Site Prefix List
    10.8.3.3/32 (Active)
    10.8.0.0/16 (Active)
    10.9.0.0/16 (Standby)
  ODE Stats Bucket Number: 1
   Last Updated  : 00:00:24 ago
    Packet Count  : 562
Byte Count    : 47208
    One Way Delay : 71 msec*
    Loss Rate Pkts: 0.0 %
    Loss Rate Byte: 0.0 %
    Jitter Mean   : 619 usec
    Unreachable   : FALSE
  ODE Stats Bucket Number: 2
   Last Updated  : 00:00:54 ago
    Packet Count  : 558
    Byte Count    : 46872
    One Way Delay : 55 msec*
    Loss Rate Pkts: 0.0 %
    Loss Rate Byte: 0.0 %
    Jitter Mean   : 556 usec
    Unreachable   : FALSE
   TCA Statistics:
      Received:133 ; Processed:133 ; Unreach_rcvd:0
  Latest TCA Bucket
   Last Updated  : 00:00:24 ago
    One Way Delay : 71 msec*
    Loss Rate Pkts: NA
    Loss Rate Byte: NA
    Jitter Mean   : NA
    Unreachability: FALSE
.
.
.
---------------------------------------------------------------------------------------------
R10# show domain default border status

---------------------------------------------------------------------------------------------

Tue Mar 24 04:52:50.379

**** Border Status ****

Instance Status: UP
Present status last updated: 3d14h ago
Loopback: Configured Loopback0 UP (10.2.10.10)
Master: 10.2.10.10
Master version: 2
Connection Status with Master: UP
MC connection info: CONNECTION SUCCESSFUL
Connected for: 3d14h
Route-Control: Enabled
Asymmetric Routing: Disabled
Minimum Mask length: 28
Sampling: off
Minimum Requirement: Met
External Wan interfaces:
Name: Tunnel100 Interface Index: 14 SNMP Index: 8 SP: MPLS Status: UP Zero-SLA: NO Path-id List: 0:10, 1:30
Name: Tunnel200 Interface Index: 15 SNMP Index: 9 SP: INET Status: UP Zero-SLA: NO Path-id List: 0:20, 1:40

Auto Tunnel information:

Name:Tunnel0 if_index: 13
Borders reachable via this tunnel:

---------------------------------------------------------------------------------------------
R10# show domain default master status   

---------------------------------------------------------------------------------------------

*** Domain MC Status ***

Master VRF: Global

Instance Type: Branch
Instance id: 0
Operational status: Up
Configured status: Up
Loopback IP Address: 10.2.10.10
Load Balancing:
Operational Status: Up
Max Calculated Utilization Variance: 1%
Last load balance attempt: never
Last Reason: Variance less than 20%
Total unbalanced bandwidth:
External links: 0 Kbps Internet links: 0 Kbps
Route Control: Enabled
Mitigation mode Aggressive: Disabled
Policy threshold variance: 20
Minimum Mask Length: 28
Minimum Requirement: Met

Borders:
IP address: 10.2.10.10
Version: 2
Connection status: CONNECTED (Last Updated 3d14h ago )
Interfaces configured:
Name: Tunnel100 | type: external | Service Provider: MPLS | Status: UP | Zero-SLA: NO
Number of default Channels: 0

Path-id list: 0:10 1:30

Name: Tunnel200 | type: external | Service Provider: INET | Status: UP | Zero-SLA: NO
Number of default Channels: 0

Path-id list: 0:20 1:40

Tunnel if: Tunnel0

---------------------------------------------------------------------------------------------
R10# show domain default master site-capability 10.9.3.3 path-id   

---------------------------------------------------------------------------------------------

Site id : 10.9.3.3
Site pop id : 1
Site mc type : Transit
Border Address : 10.9.4.4
Service provider: MPLS path-id: 30 if_index: 11
Border Address : 10.9.5.5
Service provider: INET path-id: 40 if_index: 11

---------------------------------------------------------------------------------------------
R10# show domain default master site-capability 10.8.3.3 path-id   

---------------------------------------------------------------------------------------------

Site id : 10.8.3.3
Site pop id : 0
Site mc type : Hub
Border Address : 10.8.5.5
Service provider: INET path-id: 20 if_index: 11
Border Address : 10.8.4.4
Service provider: MPLS path-id: 10 if_index: 11

---------------------------------------------------------------------------------------------
R10#  show domain default border channels service-provider INET  

---------------------------------------------------------------------------------------------

Tue Mar 24 04:53:39.968

Border Smart Probe Stats:

Smart probe parameters:
Source address used in the Probe: 10.2.10.10
Unreach time: 1000 ms
Probe source port: 18000
Probe destination port: 19000
Interface Discovery: ON
Probe freq for channels with traffic :10 secs
Discovery Probes: OFF
Number of transit probes consumed :0
Number of transit probes re-routed: 0
DSCP's using this: [0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37] [38] [39] [40] [41] [42] [43] [44] [45] [46] [47] [48] [49] [50] [51] [52] [53] [54] [55] [56] [57] [58] [59] [60] [61] [62] [63] [64]
All the other DSCPs use the default interval: 10 secs


Channel id: 6
Channel create time: 3d14h ago
Site id : 10.8.3.3
DSCP : default[0]
Service provider : INET
Pfr-Label : 0:20 | 0:0 [0x140000]
exit path-id: 0
Exit path-id sent on wire: 0
Number of Probes sent : 5657983
Number of Probes received : 5823008
Last Probe sent : 00:00:00 ago
Last Probe received : 00:00:00 ago
Channel state : Discovered and open
Channel next_hop : 10.0.200.85
RX Reachability : Reachable
TX Reachability : Reachable
Channel is sampling 0 flows
Channel remote end point: 10.0.200.85
Channel to hub: TRUE
Version: 3
Supports Zero-SLA: Yes
Muted by Zero-SLA: No
Probe freq with traffic : 1 in 10000 ms
.
.
.

---------------------------------------------------------------------------------------------
R10#  show ip nhrp nhs   
---------------------------------------------------------------------------------------------

Legend: E=Expecting replies, R=Responding, W=Waiting
Tunnel100:
10.0.100.84 RE NBMA Address: 172.16.84.4 priority = 0 cluster = 0
10.0.100.94 RE NBMA Address: 172.16.94.4 priority = 0 cluster = 0

Tunnel200:
10.0.200.85 RE NBMA Address: 172.16.85.5 priority = 0 cluster = 0
10.0.200.95 RE NBMA Address: 172.16.95.5 priority = 0 cluster = 0


---------------------------------------------------------------------------------------------