Configure Nexus Dashboard Orchestrator to Migrate Endpoint from One DC to another DC

Introduction

This document describes the design and configure changes required to migrate an Endpoint from one Data Center to another Datacenter. 

Physical Topology

Figure 1 depicts the inter connectivity of two Data Centers. 

Figure 1: Physical Topology

DC and DR locations have the Application Centric Infrastructure (ACI). DC and DR locations have the WAN Switches, Border Leaf, Spines, Inter-Site Network Devices (ISN), Server Leaf and connected Endpoints. 

Logical Topology

Figure 2: Logical Topology

Logical objects configured in both sites:

• Tenant Production is configured in DC and DR sites.
• DC-VRF-WEB and DC-VRF-APP is configured in DC-SITE1. DR-VRF-WEB and DR-VRF-APP is configured in DR-SITE2.
• Each VRF is configured with local L3Outs on Border Leaf towards WAN Switches. Default routes are configured on Border Leaf towards WAN Switches.
• WAN Switches are configured with Static routing for Inter-VRF and Inter-DC communication.
• Both Data Centers are configured with Local BDs and EPGs. DC has DC-BD1-WEB/DC-EPG1-WEB, DC-BD2-WEB/DC-EPG2-WEB and DC-BD-APP/DC-EPG-APP. DR has DR-BD1-WEB/DR-EPG1-WEB, DR-BD2-WEB/DR-EPG2-WEB and and DR-BD-APP/DR-EPG-APP.
• There are endpoints connected in WEB and APP EPG.
• DC-SITE1 and DR-SITE2 are added in Nexus Dashboard Orchestrator.

Traffic Flow before Endpoint Migration

There are multiple types of Traffic Flow in Data Centers:

• Intra EPG Traffic flow
• Inter EPG Traffic flow
• Inter VRF Traffic flow
• Inter DC traffic flow

Intra EPG Traffic Flow

Figure 3: Intra EPG Traffic Flow

Communication between DC-EP-1 and DC-EP-2 is Intra-EPG communication, as both Endpoints belongs to DC-EPG1-WEB. Communication between DR-EP-1 and DR-EP-2 is Intra-EPG communication, as both Endpoints belongs to DR-EPG1-WEB.

Inter EPG Traffic Flow

Figure 4: Inter EPG Traffic Flow

DC-EP-1 and DC-EP-3 are part of DC-EPG1-WEB and DC-EPG2-WEB respectively, communication between these two Endpoints is Inter EPG traffic Flow. DR-EP-1 and DR-EP-3 are part of DR-EPG1-WEB and DR-EPG2-WEB respectively, communication between these two Endpoints is Inter EPG traffic Flow.

Inter VRF Traffic Flow

Figure 5: Inter VRF Traffic Flow

DC Border Leaf forwards the traffic to DC WAN Switches for any Inter-VRF communication. DC WAN Switches are used for Inter-VRF communication. DC-EP-1/EP-2 (VRF WEB) communicate to DC-EP-4 (VRF APP) through WAN Switches. DR Border Leaf forwards the traffic to DR WAN Switches for any Inter-VRF communication. DR WAN Switches are used for Inter-VRF communication. DR-EP-1/EP-2 (VRF WEB) communicate to DR-P-4 (VRF APP) through WAN Switches. 

Inter DC Traffic Flow

Figure 6: Inter DC Traffic Flow

Communication between DC-Endpoints and DR-Endpoints forwarded to Border Leaf. Border Leaf forwards the traffic to WAN Switches. WAN Switches are used for Inter DC communication.

Migration Plan

Nexus Dashboard Orchestrator is used to create the Multisite between both the sites, EPGs/BDs stretched across sites and endpoints to be migrated from DC-SITE1 to DR-SITE2,

Schema-1 Creation

Schema-1 created through Nexus Dashboard Orchestrator. 

Figure 7: Tenant Template - Add Schema

Figure 8: Add Schema name

Template-VRF-Contract-Stretched Creation

Template-VRF-Contract-Stretched created inside Schema-1. DC-SITE1 and DR-SITE2 to be part of this Template and Tenant-Production to be associated with the same Template. This is stretched template. VRF and Contracts must be part of separate Template, as these objects are shared across other BD/EPGs. This template to be used to stretch the DC-SITE1 VRF and Contract to DR-SITE2.

Figure 9: Add application Template - Select ACI Multi-Cloud

Figure 10: Add Template name Template-WEB-VRF-Contract-Stretched, Select Tenant Production

Figure 11: Template-WEB-VRF-Contract-Stretched Details

Import VRF-Contract in Template-VRF-Contract-Stretched

Import DC-VRF-WEB and DC-VRF-WEB-Contract from DC-SITE1. Contracts are created for Inter-EPG communication and EPG-to-L3Out communication.

Figure 12: Click on Import and select DC-SITE1

Figure 13: Select Contract from DC-SITE1

Figure 14: Select Filter from DC-SITE1

Figure 15: Select VRF from DC-SITE1

Figure 16: Template-WEB-VRF-Contract-Stretched with VRF and Contract information

Deploy Template-VRF-Contract-Stretched

Click on Deploy Template-VRF-Contract-Stretched and select DC-SITE1 and DR-SITE2

Figure 17:Add Fabrics to Template-VRF-Contract-Stretched 

Figure 18: Deploy out Sync Templates

Figure 19: Deployment completed

Figure 20: Verify VRF and Contracts deployed on both Sites

Template-EPG1-BD1-Stretched Creation

Template-EPG1-BD1-Stretched created inside Schema-1. DC-SITE1 and DR-SITE2 added to Template and Tenant-Production  associated with the same Template. This is stretched template. This template used to stretch DC-EPG1-WEB and DC-BD1-WEB to DR-SITE2.

Figure 21: Add application Template - Select ACI Multi-Cloud

Figure 22: Add Template name Template-EPG1-BD1-Stretched, Select Tenant Production

Figure 23: Template-EPG1-BD1-Stretched Details

Import EPG1-BD1 in Template-EPG1-BD1-Stretched

Import DC-EPG1-WEB and DC-BD1-WEB from DC-SITE1.

Figure 24: Click on Import and select DC-SITE1

Figure 25: Select DC-EPG1-WEB from DC-SITE1

Figure 26: Select DC-BD1-WEB from DC-SITE1

Change BD setting in Template-EPG1-BD1-Stretched

Enable L2 Stretch in DC-BD1-WEB settings and add the Gateway IP Address. This template used to stretch BD across the site and the anycast Gateway configured in DC-SITE1 and DR-SITE2.

Figure 27: Select L2 Stretch in DC-BD1-WEB

Figure 28: Add Gateway IP/Subnet

Deploy Template-EPG1-BD1-Stretched

Click on Deploy Template-EPG1-BD1-Stretched and select DC-SITE1 and DR-SITE2

Figure 29:Add Fabrics to Template-EPG1-BD1-Stretched 

Figure 30: Deploy out Sync Templates

Figure 31: Deployment completed

Migrate DC-EP-1 from DC-SITE1 to DR-SITE2

Configure static binding in DR-SITE2 in DC-EPG1-WEB and associate DR-SITE2 Physical Domain. Migrate the DC-EP-1 from DC-SITE1 to DR-SITE2.

Figure 32: DC-EP-1 currently learned in DC-SITE1

Figure 33: DC-EP-1 removed from DC-SITE1

Figure 34: Adding Physical domain in DR-SITE2

Figure 35: Adding Static Binding in DR-SITE2

Figure 36: DC-EP-1 learned in DR-SITE2

Physical Design after DC-EP-1 Migration

DC-EP-1 is connected to DR-SITE2 Server Leaf. 

Figure 37: Physical Design after DC-EP-1 Migration

Logical Design after DC-EP-1 Migration

DC-EP-1 is connected to DR-SITE2 Server Leaf. DC-EPG1-WEB, DC-BD1-WEB and DC-VRF-WEB are stretched between DC-SITE1 and DR-SITE2.

Figure 38: Logical Design after DC-EP-1 Migration

Intra EPG Traffic Flow after DC-EP-1 Migration

Figure 39: Intra EPG Traffic Flow after DC-EP-1 Migration

Communication between DC-EP-1 and DC-EP-2 is Intra-EPG communication, as both Endpoints belongs to DC-EPG1-WEB. This communication happens through DC ISN to DR ISN Multisite/Overlay Links. 

Ping response between DC-EP-1 and DC-EP-2

Figure 40: Ping response between DC-EP-1 and DC-EP-2

Routing Table from Spines

DC-EP-1 learned in DC-SP-01/DC-SP-02 from DR-SP-01/DR-SP-02.

Figure 41: Routing Table from Spines

DC-EP-1 is learned in DC-SITE1-SP-01 from DR-SITE2-SP-01

DR-SITE2-SP-01 Overlay Unicast TEP IP

Template-EPG2-BD2-Site1 Creation

Inter EPG communication between DC-EP-1 and DC-EP-3 happens, once DC-EPG2-WEB and DC-BD2-WEB  are part of Nexus Dashboard Orchestrator.

Template-EPG2-BD2-Site1 created inside Schema-1. DC-SITE1 added to Template and Tenant-Production  associated with the same Template. This is site specific template. This template used to import the Template-EPG2-BD2-Site1 for the communication between DC-EP-1 and DC-EP-3.

DC-EP-1 and DC-EP-3 communication requires DC-EPG2-BD2 has to be part of Nexus Dashboard Orchestrator.

Figure 42: DC-EP-1 and DC-EP-3 not able to communicate

Figure 43: Add application Template - Select ACI Multi-Cloud

Figure 44: Add Template name Template-EPG2-BD2-Site1, Select Tenant Production

Figure 45: Template-EPG2-BD2-Site1 Details

Import EPG2-BD2 in Template-EPG2-BD2-Site1

Import DC-EPG2-WEB and DC-BD2-WEB from DC-SITE1.

Figure 46: Click on Import and select DC-SITE1

Figure 47: Select DC-EPG2-WEB from DC-SITE1

Figure 48: Select DC-BD2-WEB from DC-SITE1

Figure 49: Contract associated with DC-EPG2-WEB are imported

Deploy Template-EPG2-BD2-Site1

Click on Deploy Template-EPG2-BD2-Site1 and select DC-SITE1

Figure 50: Add Fabrics to Template-EPG2-BD2-Site1

Figure 51: Deploy out Sync Templates

Figure 52: Deployment completed

Figure 53: DC-EPG2-WEB is deployed in both sites

Shadow EPG for DC-EPG2-WEB created in DR-SITE2

Inter EPG Traffic Flow after EP-1 Migration

Figure 54: Inter EPG Traffic Flow after EP-1 Migration

Communication between DC-EP-1 and DC-EP-3 is Inter-EPG communication, as both Endpoints belongs to DC-EPG1-WEB and DC-EPG2-WEB respectively. This communication happens through DC ISN to DR ISN Multisite/Overlay Links. 

Ping response between DC-EP-1 and DC-EP-3

Figure 55: Ping response between DC-EP-1 and DC-EP-3

Template-WEB-L3Out-Site1 Creation

Template-Web-L3Out-Site1 created inside Schema-1. DC-SITE1 added to template and Tenant-Production associated with the same Template. This is site specific template. This template used for DC-EP-1 Inter-VRF and Inter-DC communication.

Figure 56: Add application Template - Select ACI Multi-Cloud

Figure 57: Add Template name Template-WEB-L3Out-Site1, Select Tenant Production

Figure 58: Template-WEB-L3Out-Site1 Details

Import External EPG and L3Out in Template-WEB-L3Out-Site1

Import External EPG and L3Out in Template-WEB-L3Out-Site1

Figure 59: Click on Import and select DC-SITE1

Figure 60:Select EXT-APP-EPG from DC-SITE1

Figure 61: Select DC-APP-L3Out from DC-SITE1

Figure 62: Contract associated with EXT-WEB-EPG are imported

Shadow of EXT-WEB-EPG created in DR-SITE2 with applied DC contracts.

Deploy Template-WEB-L3Out-Site1

Click on Deploy Template-WEB-L3Out-Site1 and select DC-SITE1

Figure 63: Add Fabrics to Template-WEB-L3Out-Site1

Figure 64:Deploy out Sync Templates

Figure 65: Deployment completed

Verify the routes in DR Server Leaf for DC-VRF-WEB

Static routes installed in DR Server Leaf for DC-VRF-WEB.

Figure 66: Verify the routes in DR Server Leaf for DC-VRF-WEB

Inter VRF Traffic Flow after DC-EP-1 Migration

Figure 67: Inter VRF Traffic Flow after DC-EP-1 Migration

DC-EP-1 uses DC-WEB-L3Out to communicate with DC-EP-4. The traffic flows from DR-ISN to DC-ISN Multisite Links, DC-ISN to DC-SP-01/DC-SP-02 and from DC-SP to DC-BL. DC-BL-01/DC-BL-02 forward the traffic to DC-WAN Switches for Inter-VRF routing.

Ping response between DC-EP-1 and DC-EP-4

Figure 68: Ping response between DC-EP-1 and DC-EP-4

Inter DC Traffic Flow after DC-EP-1 Migration

Figure 69: Inter DC Traffic Flow after DC-EP-1 Migration

DC-EP-1 uses DC-WEB-L3Out to communicate with DR Endpoints. The traffic flows from DR-ISN to DC-ISN Multisite Links, DC-ISN to DC-SP-01/DC-SP-02 and from DC-SP to DC-BL. DC-BL-01/DC-BL-02 forward the traffic to DC-WAN Switches for DR Endpoints.

Ping response between DC-EP-1 and DR-EPs

Figure 70: Ping response between DC-EP-1 and DR-EPs

Migrate remaining Endpoints

Physical Design after remaining Endpoints Migration

After migrating the remaining Endpoints from DC to DR DC-EPG1-WEB, physical diagram changed accordingly.

Figure 71: Physical Design after all Endpoints Migration from DC to DR

Logical Design after remaining Endpoints Migration

DC-EPG1-WEB, DC-BD1-WEB and DC-VRF-WEB are already stretched between DC and DR Sites. DC remaining Endpoints migrated from DC to DR Site. 

Figure 72: Logical Design after remianing Endpoint Migration

Intra EPG Traffic Flow after remaining Endpoint Migration

Figure 73: Intra EPG Traffic Flow after remaining Endpoint Migration

Communication between DC-EP-1 and DC-EP-2 is Intra-EPG communication, as both Endpoints belongs to DC-EPG1-WEB. This communication happens directly within DR Site. 

Inter EPG, Inter VRF and Inter DC traffic flows remain similar to DC-EP-1 migration.

Undeploy Template-EPG1-BD1-Stretched from DC Site

All the Endpoints are migrated from DC to DR site for DC-EPG1-WEB. DC-EPG1-WEB and DC-BD1-WEB are not required in DC Site. Undeploy the Template-EPG1-BD1-Stretched from DC Site, this deletes the EPG and BD from Site-1.

Figure 74: Click on Undeploy Template

Figure 75: Select DC-SITE1 and Click undeploy

Dissociate Template-EPG1-BD1-Stretched from DC Site

This step dissociates the Template-EPG1-BD-Stretched from DC Site.

Figure 76: Click on Dissociate Template

Figure 77: Uncheck DC-SITE1

Figure 78: DC-SITE2 part of Template-EPG1-BD1-Stretched

Logical Design after Undeploying the Template-EPG1-BD1-Stretched from DC

DC-EPG1-WEB and DC-BD1-WEB is not part of DC Site after Undeploying the Template.

Figure 79: Logical Design after Undeploying the Template

Template-VRF-Contract-Site2 Creation

Template-VRF-Contract-Site2 created inside Schema-1. DR-SITE2 added to Template and Tenant-Production associated with the same Template. This is site specific template. This template used to associate VRF and Contract from DR site for DC-EPG1-WEB and DC-BD1-WEB.

Figure 80:Add application Template - Select ACI Multi-Cloud

Figure 81: Add Template name Template-VRF-Contract-Site2, Select Tenant Production

Figure 82: Template-VRF-Contract-Site2 Details

Import VRF-Contract in Template-VRF-Contract-Site2

Import DR-VRF-WEB and DR-VRF-WEB-Contract from DR-SITE2.

Figure 83:Click on Import and select DR-SITE2

Figure 84: Select Contract from DR-SITE2

Figure 85: Select Filter from DR-SITE2

Figure 86: Select VRF from DR-SITE2

Figure 87: Template-WEB-VRF-Contract-Site2 with VRF/Contract information

Deploy Template-VRF-Contract-Site2

Click on Deploy Template-VRF-Contract-Site2  and select DR-SITE2

Figure 88:Add Fabrics to Template-VRF-Contract-Site2

Figure 89: Deploy out Sync Templates

Figure 90: Deployment completed

Associate DR-VRF-WEB to DC-BD1-WEB

Associate DR-VRF-WEB to DC-BD1-WEB from Template-EPG1-BD1-Stretched which was created earlier. DC-BD1-WEB is part of DR-SITE2.

Figure 91: Click on Template-EPG1-BD1-Stretched

Figure 92: Associate DR-VRF-WEB to DC-BD1-WEB

Apply DR-Contracts to DC-EPG1-WEB

Apply DR-Contract to DC-EPG1-WEB which uses DR contracts for the communication from DC-EPG1-WEB for Inter-DC, Inter-VRF and Inter-EPG. DC-EPG1-WEB is part of DR-SITE2

Figure 93: Delete DC-Contracts from DC-EPG1-WEB

Figure 94: Add DR-Contracts in DC-EPG1-WEB

Figure 95: Template-EPG1-BD1-Stretched information

Figure 96: Deploy out Sync Templates

Figure 97: Deployment completed

DC-Endpoint-1 Traffic Flow

DC-Endpoint-1 starts using DR-L3Out-WEB for the communication with DC Endpoints. This communication requires necessary routing changes on WAN Switches.

Figure 98: DC-Endpoint-1 Traffic Flow

Ping response between DC-EP-1 and DC/DR-EPs

Figure 99: Ping response between DC-EP-1 and DC-EP-2