PfRv3 Inter-DC Optimization

The PfRv3 Inter-DC (IDC) Optimization feature optimizes traffic between hub and transit hub sites over a WAN overlay or a DCI overlay. A path-preference policy specific to inter-DC Optimization is used for optimizing traffic between two or more hub sites. The PfRv3-Inter-DC-Optimization routs traffic from a hub site to another hub site for specific traffic types such as data, voice, video, and so on.

Feature Information for PfRv3 Inter-DC Optimization

The following table provides release information about the feature or features described in this module. This table lists only the software release that introduced support for a given feature in a given software release train. Unless noted otherwise, subsequent releases of that software release train also support that feature.

Use Cisco Feature Navigator to find information about platform support and Cisco software image support. To access Cisco Feature Navigator, go to www.cisco.com/go/cfn. An account on Cisco.com is not required.
Table 1. Feature Information for PfRv3 Inter-DC Optimization

Feature Name

Releases

Feature Information

PfRv3 Inter-DC Optimization

Cisco IOS XE Everest 16.6.1

The following commands were introduced or modified: domain, inter-dc, interdc-path-preference.

Prerequisites for PfRv3 Inter-DC Optimization

  • Hub sites must be upgraded for using the same version of IOS for the master and border devices.

  • Static NHRP mapping must be used between hub sites. (NHRP shortcuts are not allowed between hub sites)

  • Local LAN prefixes on each hub site (all borders) must have a specific route pointing to LAN interfaces and not to DCI or WAN interfaces.

Limitations and Guidelines for Inter-DC Optimization

  • The PfRv3 Inter-DC Optimization does not optimize routes using common prefixes.


    Note

    A common prefix is a prefix which is configurared as a static prefix on all the hub sites, that include hub sites and transite hub sites.

  • The command domain dci-path should be added in DCI tunnel interface, but normal WAN interface with domain path command can also be chosen as DCI path. But DCI interface using domain dci-path cannot be chosen as the path for normal hub to spoke traffic.

  • We recommend to use static configuration under DCI tunnel interface to set up peer between DC sites. If nhrp shortcut is used, a forwarding loop may occur.

  • After enabling the IDC feature using the inter-dc command, you can configure path-preference and interdc-path-preference under policy.


    Note

    You should not configure DCI path in global path-preference because if you add DCI path into path-preference, there is no channel available between hub and spoke in the DCI path. The DCI path cannot be chosen for the normal traffic-classes.

  • The IDC feature must be enabled on both peer masters. It is recommended to use the same overlay routing protocol for all WAN and DCI tunnels.

Information About PfRv3-Inter-DC-Optimization

Datacenter Optimization

The following figure illustrates the PfRv3 Inter-DC Optimization feature where traffic between hub sites DC1, DC2 and DC3 are routed to forward specific traffic trough a specific hub. The figure shows four paths can be used as candidates for the traffic from DC1 to DC2. IDC1 and IDC2 are Inter-DC links those can be used for this traffic. MPLS and INET are normal WAN paths that can also be used for this traffic as candidates. It depends on the path-preference policy specific to inter-DC optimization.

Figure 1. Datacenter Optimization

The PfRv3 Inter-DC Optimization feature can be enabled with the inter-dc command in domain master controller advanced mode. All hubs in the network must be connected through WAN overlay or DCI overlay. All hub and transit hub masters must be enabled with this feature locally. WAN overlay is configured by defining a WAN interface using the domain path command. DCI overlay is configured by defining a DCI interface using the domain dci-path command.

The salient points of the PfRv3 Inter-DC Optimization feature are as follows:

  • The domain dci-path command enables route control which routes the transit traffic on all DCI interfaces in ingress direction.

  • Traffic classes are learnt based on the egress aggregate update and traffic channels over the WAN and DCI overlay.

  • Tunnel addresses and path ID mapping are exchanged by site capability between the hub and transit masters.


    Note

    The tunnel IP address for corresponding interface or path-id is advertised among the hub and transit masters when the PfRv3 Inter-DC Optimization feature is enabled.

DCI Path Options

Based on the actual deployment requirement, you can choose any of the following options for providing the DCI path:

Using the existing DMVPN overlay and the same tunnel interface:

In the hub to spoke DMVPN tunnel interface configuration, there is no dmvpn peer between DC sites. So, if the normal hub tunnel interface is used as DCI path, some additional configuration should be added to set up the dmvpn peer between DC sites, such as ip nhrp nhs 161.1.0.5 nbma 155.155.155.5 multicast above.

Using an independent DCI link(s) with independent DMVPN overlay

When there is dedicated DCI links between DC sites, a dedicated DMVPN overlay can be used as DCI path. And ideally, the dedicated DCI links are more stable than the normal WAN links. Using the existing hub to spoke DMVPN, or using a dedicated DCI DMVPN built over dedicated DCI links will depend on the available interfaces in the network, and which solution will meet the need of the network


Note

A third option of building a second set of DMVPN tunnels using the same transport as the existing DMVPN hub and spoke network is not recommended and it has not been validated.

How to Configure PfRv3-Inter-DC-Optimization

Specifying the DCI interface on a Hub Site


enable
configure terminal
interface tunnel155
 domain dci-path DCI path-id 11
   exit

Configuring Inter-DC on Hub Master Controller

To configure the Inter-DC Optimization feature on the hub master controller, use the following commands:

enable
configure terminal
domain default
 vrf green
  master hub
   source-interface Loopback1
   site-prefixes prefix-list HUBPFX
   advanced
    inter-dc
     enterprise-prefix prefix-list ENTPFX
     class BUSINESS sequence 10
      match dscp ef policy custom
      priority 1 one-way-delay threshold 100
    interdc-path-preference DCI1 DCI2 fallback MPLS next-fallback INET
    exit

Configuring Inter-DC on Transit Hub

To configure Inter-DC on the transit hub, use the following commands:

enable
Configure terminal
domain default
 vrf green
  master transit 2
   source-interface Loopback1
   site-prefixes prefix-list HUBPFX
   hub 100.10.1.1
   advanced
inter-dc 
  class BUSINESS sequence 10
    interdc-path-preference DCI1 fallback MPLS next-fallback INET
exit

Specifying IDC Local Policy

This is an optional task to overwrite the global path-preference.


enable
configure termainal
domain default
 vrf green
  master transit 2
   class BUSINESS sequence 10
    interdc-path-preference DCI1 fallback ISP1 next-fallback ISP2
exit

Verifying Inter-DC Configuration


HMCBR# show domain default vrf green master status
  *** Domain MC Status ***
 Master VRF: green
  Instance Type:    Hub
  Instance id:      1
  Operational status:  Up
  Configured status:  Up
  Loopback IP Address: 100.10.1.1
  Global Config Last Publish status: Peering Success
  Smart Probe Profile:
    General Monitor:
      Packets per burst: 1
      Interval(secs): 1
    Quick Monitor:
      Packets per burst: 20
      Interval(secs): 1
  Load Balancing:
   Admin Status: Disabled
   Operational Status: Down
   Enterprise top level prefixes configured: 0
  Route Control: Enabled
  Transit Site Affinity: Enabled
  Load Sharing: Enabled
  Path Pruning Depth: Disabled
  Inter-DC Optimization: Enabled

Verifying Master Controller Configuration

HMCBR# show domain default vrf green master status

  *** Domain MC Status ***
 Master VRF: green
  Instance Type:    Hub
  Instance id:      1
  Operational status:  Up
  Configured status:  Up
  Loopback IP Address: 100.10.1.1
  Global Config Last Publish status: Peering Success
..... ..... 
Minimum Packet Loss Calculation Threshold: 15 packets
  Minimum Bytes Loss Calculation Threshold: 1 bytes
  Borders:
    IP address: 100.10.3.1
    Version: 2
    Connection status: CONNECTED (Last Updated 15:44:28 ago )
    Interfaces configured:
      Name: Tunnel10 | type: external | Service Provider: ISP1 path-id:3 | Status: UP | Zero-SLA: NO | Path of Last Resort: Disabled
          Number of default Channels: 0
          
      Name: Tunnel40 | type: external | Service Provider: ISP4 path-id:9 | Status: UP | Zero-SLA: NO | Path of Last Resort: Disabled
          Number of default Channels: 0
      Name: Tunnel155 | type: DCI | Service Provider: DCI1 path-id:103 | Status: UP | Zero-SLA: NO | Path of Last Resort: Disabled
          Number of default Channels: 0
    Tunnel if: Tunnel0
    IP address: 100.10.1.1
    Version: 2
    Connection status: CONNECTED (Last Updated 15:44:21 ago )
    Interfaces configured:
      Name: Tunnel10 | type: external | Service Provider: ISP1 path-id:1 | Status: UP | Zero-SLA: NO | Path of Last Resort: Disabled
          Number of default Channels: 0
      Name: Tunnel30 | type: external | Service Provider: ISP3 path-id:7 | Status: UP | Zero-SLA: NO | Path of Last Resort: Disabled
          Number of default Channels: 0
      Name: Tunnel155 | type: DCI | Service Provider: DCI1 path-id:101 | Status: UP | Zero-SLA: NO | Path of Last Resort: Disabled
          Number of default Channels: 0

Verifying the Channel Status


HMCBR# show domain default vrf green master channels
   Legend: * (Value obtained from Network delay:)
Channel Id: 7  Dst Site-Id: 100.20.1.1  Link Name: ISP1  DSCP: default [0] pfr-label: 0:0 | 0:3 [0x3] TCs: 0  BackupTCs: 0
  Channel Created: 15:43:53 ago
  Provisional State: Initiated and open
  Operational state: Available
  Channel to hub: FALSE
  Inter-DC Channel: FALSE
  Interface Id: 25 
  Supports Zero-SLA: Yes
  Muted by Zero-SLA: No
  Unreach Probing only: OFF
  Estimated Channel Egress Bandwidth: 0 Kbps
  Immitigable Events Summary: 
   Total Performance Count: 0, Total BW Count: 0
   ODE Statistics:
      Received: 0
   TCA Statistics:
      Received: 0 ; Processed: 0 ; Unreach_rcvd: 0 ; Local Unreach_rcvd: 0
      TCA lost byte rate: 0
      TCA lost packet rate: 0
      TCA one-way-delay: 0
      TCA network-delay: 0
      TCA jitter mean: 0
Channel Id: 117  Dst Site-Id: 100.16.1.1  Link Name: ISP1  DSCP: default [0] pfr-label: 3:13 | 0:3 [0x30D0003] TCs: 0  BackupTCs: 0
  Channel Created: 15:33:02 ago
  Provisional State: Initiated and open
  Operational state: Available
  Channel to hub: TRUE
  Inter-DC Channel: TRUE
  Interface Id: 25 
  Supports Zero-SLA: Yes
  Muted by Zero-SLA: No
  Unreach Probing only: OFF
  Estimated Channel Egress Bandwidth: 0 Kbps
  Immitigable Events Summary: 
   Total Performance Count: 0, Total BW Count: 0
  Site Prefix List
    100.16.1.1/32 (Routable)
    100.16.0.0/16 (Routable)
    100.10.0.0/16 (Routable)
    100.15.0.0/16 (Routable)
    100.0.0.0/8 (Routable)
   ODE Statistics:
      Received: 0
   TCA Statistics:
      Received: 0 ; Processed: 0 ; Unreach_rcvd: 0 ; Local Unreach_rcvd: 0
      TCA lost byte rate: 0
      TCA lost packet rate: 0
      TCA one-way-delay: 0
      TCA network-delay: 0
      TCA jitter mean: 0

Example Configurations for PfRv3 Inter-DC

Example for Policy Configured on the Hub MC with Inter DC

In this example, the policy can work on the normal hub-spoke traffic and the IDC traffic. For IDC traffic, the ‘interdc-path-preference’ takes effect. DCI1 and DCI2 are primary paths. If they are out-of-policy, the MPLS, which is a backup path, will be used. For normal hub-spoke traffic, the ‘path-preference’ takes effect. The other configuration is same as normal PfRv3 policy. For example, the threshold of delay is 100 ms for both the IDC traffic and the normal hub-spoke traffic. 

class BUSINESS sequence 10
    match dscp ef policy custom
    priority 1 one-way-delay threshold 100
    path-preference MPLS fallback INET
    interdc-path-preference DCI1 DCI2 fallback MPLS next-fallback INET

Example for Policy Configured on the Transit Hub MC with Inter DC

On transit hub master, you can see the same policy. But, if the interdc-path-preference is configured on this transit hub. The local interdc-path-preference will overwrite the policy from hub site 

class BUSINESS sequence 10
    interdc-path-preference DCI1 fallback MPLS next-fallback INET

Example for 'show domain vrf master policy' on hub master

--------------------------------------------------------------------------------
Global-policy-list:

  class BUSINESS sequence 10
    path-preference MPLS fallback INET
    class type: Dscp Based
      match dscp ef policy custom
        priority 1 one-way-delay threshold 100 msec
--------------------------------------------------------------------------------
InterDC-policy-list:

  class BUSINESS sequence 10
interdc-path-preference DCI1 DCI2 fallback MPLS next-fallback INET
class type: Dscp Based
      match dscp ef policy custom
        priority 1 one-way-delay threshold 100 msec