With Cisco Catalyst SD-WAN multitenancy, a service provider can manage multiple customers, called tenants, from Cisco SD-WAN Manager. The tenants share the same set of underlying Cisco SD-WAN controllers: Cisco SD-WAN Manager, Cisco Catalyst SD-WAN Validator, and Cisco Catalyst SD-WAN Controller. The tenant data is logically isolated on these shared controllers.

The service provider accesses Cisco SD-WAN Manager using a domain name mapped to the IP address of a Cisco SD-WAN Manager cluster and manages the multitenant deployment. Each tenant is provided a subdomain to access a tenant-specific Cisco SD-WAN Manager view and manage the tenant deployment. For example, a service provider using the domain name managed-sp.com, can assign tenants Customer1 and Customer2 the subdomains customer1.managed-sp.com and customer2.managed-sp.com and manage them on the same set of Cisco SD-WAN controllers, instead of providing each customer a single-tenant setup with a dedicated set of Cisco SD-WAN controllers.

Following are the key features of Cisco Catalyst SD-WAN multitenancy:

• Full enterprise multitenancy: Cisco Catalyst SD-WAN supports multitenancy and offers enterprises the flexibility of segregated roles such as service provider and tenants. Service providers can use multitenancy to provide Cisco Catalyst SD-WAN service offerings to their customers.

• Multi-tenant Cisco SD-WAN Manager

• Multi-tenant Cisco Catalyst SD-WAN Validators

• Multi-tenant Cisco Catalyst SD-WAN Controllers

• Tenant-specific WAN Edge Devices

• Overlapping VPN numbers: A particular VPN or a set of common VPNs is assigned to a specific tenant, with their own configurations and monitoring dashboard environment. These VPN numbers can overlap where they are used by other tenants.

• On-prem and cloud deployment models: Cisco Catalyst SD-WAN controllers can be deployed in an organization data center on servers running the VMware ESXi 6.7 or later, or the Kernel-based Virtual Machine (KVM) hypervisor. Cisco Catalyst SD-WAN controllers can also be hosted on Amazon Web Services (AWS) servers by Cisco CloudOps.

• Tenant-specific Cisco SD-WAN Analytics: Cisco SD-WAN Analytics is a cloud-based service that offers insights into the performance of applications and the underlying SD-WAN network infrastructure. Each tenant can obtain Cisco SD-WAN Analytics insights for their overlay network by requesting a tenant-specific Cisco SD-WAN Analytics instance and enabling data collection on Cisco SD-WAN Manager. The service provider must enable cloud services on Cisco SD-WAN Manager in the provider view to facilitate the onboarding of the Cisco SD-WAN Analytics instance for the tenant overlay network.

Note	Starting from Cisco SD-WAN Manager Release 20.13.1, a single node multitenancy is supported. Expand a single node cluster into a multitenant deployment.

Multi-tenant Cisco SD-WAN Manager

Cisco SD-WAN Manager is deployed and configured by the service provider. The provider enables multitenancy and creates a Cisco SD-WAN Manager cluster to serve tenants. Only the provider can access a Cisco SD-WAN Manager instance through the SSH terminal.

Cisco SD-WAN Manager offers service providers an overall view of the SD-WAN multi-tenant deployment and allows a provider to manage the shared Cisco Catalyst SD-WAN Validator and Cisco Catalyst SD-WAN Controller devices. Cisco SD-WAN Manager also allows service providers to monitor and manage the deployments of each tenant.

Cisco SD-WAN Manager allows tenants to monitor and manage their deployment. Through Cisco SD-WAN Manager, tenants can deploy and configure WAN edge devices. Tenants can also configure custom policies on assigned Cisco Catalyst SD-WAN Controllers.

Multi-tenant Cisco Catalyst SD-WAN Validators

Cisco Catalyst SD-WAN Validators are deployed and configured by the service provider. Only the provider can access a Cisco Catalyst SD-WAN Validator through the SSH terminal.

Cisco Catalyst SD-WAN Validators serve WAN edge devices of multiple tenants as the devices are added to the overlay network.

Multi-tenant Cisco Catalyst SD-WAN Controllers

Cisco Catalyst SD-WAN Controllers are deployed by the service provider. Only the provider can create and attach device and feature templates to Cisco Catalyst SD-WAN Controllers, and can access a Cisco Catalyst SD-WAN Controller through the SSH terminal.

• When a tenant is created, Cisco SD-WAN Manager assigns two Cisco Catalyst SD-WAN Controllers for the tenant. The Cisco Catalyst SD-WAN Controllers form an active-active cluster.

  Each tenant is assigned only two Cisco Catalyst SD-WAN Controllers. Before a tenant is created, two Cisco Catalyst SD-WAN Controllers must be available to serve the tenant.

• When more than one pair of Cisco Catalyst SD-WAN Controllers are available to serve a tenant, Cisco SD-WAN Manager assigns to the tenant the pair of Cisco Catalyst SD-WAN Controllers connected to the lowest number of forecast devices. If two pairs of Cisco SD-WAN Controllers are connected to the same number of devices, Cisco SD-WAN Manager assigns to the tenant the pair of Cisco SD-WAN Controllers serving the lowest number of tenants.

• From Cisco vManage Release 20.9.1, while onboarding a tenant to a multitenant deployment, you can choose the pair of multitenant Cisco SD-WAN Controllers that serve the tenant. After onboarding a tenant, you can migrate the tenant to a different pair of multitenant Cisco SD-WAN Controllers, if necessary. For more information, see Flexible Tenant Placement on Multitenant Cisco SD-WAN Controllers.

• Each pair of Cisco Catalyst SD-WAN Controllers can serve a maximum of 24 tenants.

• Tenants can configure custom policies on the Cisco Catalyst SD-WAN Controllers assigned to them. Cisco SD-WAN Manager notifies the Cisco Catalyst SD-WAN Controllers to pull the policy templates. Cisco Catalyst SD-WAN Controllers pull the templates and deploy the policy configuration for the specific tenant.

• Only the provider can view events, audit logs, and OMP alarms for a Cisco Catalyst SD-WAN Controller on Cisco SD-WAN Manager.

Tenant-Specific WAN Edge Devices

A tenant or the provider acting on behalf of a tenant can add WAN edge devices to the tenant network, configure the devices, and remove the devices from the tenant network, or access the device through the SSH terminal.

A provider can manage the WAN edge devices only from provider-as-tenant view. In the provider view, Cisco SD-WAN Manager does not show any WAN edge device information.

Cisco SD-WAN Manager reports WAN edge device events, logs, and alarms only in the tenant and the provider-as-tenant views.

Restrictions for Migration of a Tenant from a Multitenant Overlay to a Single-Tenant Deployment

Minimum supported releases: Cisco IOS XE Catalyst SD-WAN Release 17.13.1a and Cisco Catalyst SD-WAN Manager Release 20.13.1

• Change in the tenant organization name is not supported when the tenant moves from the Cisco Catalyst SD-WAN source to destination deployment.

• Tenant migration with multitenant WAN edge devices is not supported.

• Data traffic loss is expected during migration as devices are migrating from one set of Cisco SD-WAN Controllers to another.

• All user passwords are set to the default Cisco password on the destination overlay. The default password is Cisco#123@Viptela.

• Statistical data of the tenant that can be relearned by destination Cisco SD-WAN Manager is not migrated.

• The migration procedure does not support multiple imports on the same destination Cisco SD-WAN Manager. Reinitialize the destination Cisco SD-WAN Manager to allow import again.

Prerequisites

• Download and install software versions as recommended in the following table:

  Table 4. Minimum Software Prerequisites for Cisco Catalyst SD-WAN Multitenancy

  Device	Software Version
  Cisco SD-WAN Manager	Cisco vManage Release 20.6.1
  Cisco SD-WAN Validator	Cisco SD-WAN Release 20.6.1
  Cisco SD-WAN Controller	Cisco SD-WAN Release 20.6.1
  Cisco IOS XE Catalyst SD-WAN Device	Cisco IOS XE Catalyst SD-WAN Release 17.6.1a

  A configuration in which one or more controllers, or WAN edge devices, are running software versions earlier than those mentioned in the table above is not supported.

• Do not migrate an existing single-tenant Cisco SD-WAN Manager instance into multitenant mode, even if you invalidate or delete all devices from the existing Cisco SD-WAN Manager instance. Instead, download and install a new Cisco SD-WAN Manager software image.

  Note	After you enable Cisco SD-WAN Manager for multitenancy, you cannot migrate it back to single tenant mode.

• Follow the recommended hardware specifications in the Supported Devices and Controller Specifications section of this document.

• Log in to Cisco SD-WAN Manager as the provider admin user.

1. Create Cisco SD-WAN Manager cluster.

   1. To support 50 tenants and 1000 devices across all tenants, create a 3-node Cisco SD-WAN Manager Multitenant cluster.

   2. To support 100 tenants and 5000 devices across all tenants, create a 6-node Cisco SD-WAN Manager Multitenant cluster.

   3. From Cisco IOS XE Release 17.6.3a, Cisco vManage Release 20.6.3, to support 150 tenants and 7500 devices across all tenants, create a 6-node Cisco SD-WAN Manager Multitenant cluster.

2. Create and configure Cisco SD-WAN Validator instances. See Deploy Cisco SD-WAN Validator.

   While configuring Cisco SD-WAN Validator instances, configure the service provider organization name (sp-organization-name) and the organization name (organization-name). See Configure Organization Name in Cisco SD-WAN Validator.

   sp-organization-name multitenancy
   organization-name multitenancy

3. Create Cisco SD-WAN Controller instances. See Deploy the Cisco SD-WAN Controller.

   • To support 50 tenants and 1000 devices across all tenants, deploy 6 Cisco SD-WAN Controller instances.

   • To support 100 tenants and 5000 devices across all tenants, deploy 10 Cisco SD-WAN Controller.

   • From Cisco IOS XE Release 17.6.3a, Cisco vManage Release 20.6.3, to support 150 tenants and 7500 devices across all tenants, deploy 16 Cisco SD-WAN Controllers.

   1. Add Cisco SD-WAN Controllers to the overlay network.

4. Onboard new tenants. See Add a New Tenant.

As a service provider, suppose you have deployed a C to the overlay to support up to 100 tenants and 5000 devices. From Cisco IOS XE Release 17.6.3a, Cisco vManage Release 20.6.3, you can expand the Cisco SD-WAN Manager cluster and add additional Cisco SD-WAN Controllers to the overlay to support up to 150 tenants and 7500 devices.

Prerequisites

A multitenant Cisco Catalyst SD-WAN overlay that supports up to 50 tenants and 1000 devices, deployed according to the steps outlined in the Initial Setup for Multitenancy section of this document.

1. Expand the existing 3-node Cisco SD-WAN Manager cluster to a 6-node cluster.

2. To support up to 100 tenants and 5000 devices, you must have 10 Cisco SD-WAN Controllers in the overlay. So, deploy 4 Cisco SD-WAN Controllers in addition to the 6 existing Cisco SD-WAN Controllers in the overlay.

   To support up to 150 tenants and 7500 devices, you must have 16 Cisco SD-WAN Controllers in the overlay. So, deploy 10 Cisco SD-WAN Controllers in addition to the 6 existing Cisco SD-WAN Controllers in the overlay.

   1. Create Cisco SD-WAN Controller instances. See Deploy the Cisco SD-WAN Controller.

   2. Add Cisco SD-WAN Controllers to the overlay network.

You can now add more tenants or allow your existing tenants to add more devices subject to the relevant limits.

Note	Starting from Cisco SD-WAN Manager Release 20.13.1, you can expand a single node cluster into 3 or 6 node clusters.

Tenant Device Forecasting

While adding a new tenant to the multitenant Cisco Catalyst SD-WAN deployment, a service provider can forecast the number of WAN edge devices that the tenant may deploy in their overlay network. Cisco SD-WAN Manager enforces this forecast limit. If the tenant tries to add devices beyond this limit, Cisco SD-WAN Manager responds with an appropriate error message and the device addition fails.

In a multitenant deployment, a tenant can add a maximum of 1000 devices to their overlay network.

Note	From Cisco IOS XE Release 17.6.2, Cisco vManage Release 20.6.2, you can modify the device forecast for a tenant after the tenant is added. This modification is not supported in Cisco IOS XE Release 17.6.1a, Cisco vManage Release 20.6.1.

Benefits:

• The service provider can ensure that the Cisco Catalyst SD-WAN controller resources are used more efficiently.

• Depending on the configuration, a multitenant deployment can support a fixed number of WAN edge devices across all tenants. By forecasting the number of devices a tenant may add, the service provider can assign a quota for each tenant from the overall pool of edge devices that the deployment can support.

Before You Begin

• Before you begin the migration,

  • Migration of a single-tenant overlay to a multitenant deployment is only supported with the Cisco Catalyst SD-WAN controllers deployed on-premises. Migration is yet to be supported with cloud-hosted Cisco Catalyst SD-WAN controllers.

  • Ensure that the edge devices in the single-tenant deployment can reach the Cisco SD-WAN Validator in the multitenant deployment

  • Ensure that the template, routing, and policy configuration on the edge devices is synchronized with the current configuration on Cisco SD-WAN Manager

  • Configure a maintenance window for the single-tenant overlay before performing this procedure. See Configure or Cancel SD-WAN Manager Server Maintenance Window.

• Minimum software requirements for the single-tenant overlay to be migrated:

  Device	Software Version
  Cisco SD-WAN Manager	Cisco vManage Release 20.6.1
  Cisco SD-WAN Validator	Cisco SD-WAN Release 20.6.1
  Cisco Catalyst SD-WAN Controller	Cisco SD-WAN Release 20.6.1
  Cisco IOS XE Catalyst SD-WAN device	Cisco IOS XE Catalyst SD-WAN Release 17.6.1a

• Minimum software requirements for the multitenant deployment to which the single-tenant overlay must be migrated:

  Device	Software Version
  Cisco SD-WAN Manager	Cisco vManage Release 20.6.1
  Cisco SD-WAN Validator	Cisco SD-WAN Release 20.6.1
  Cisco Catalyst SD-WAN Controller	Cisco SD-WAN Release 20.6.1
  Cisco IOS XE Catalyst SD-WAN device	Cisco IOS XE Catalyst SD-WAN Release 17.6.1a

• The software versions of the Cisco Catalyst SD-WAN controllers and WAN edge devices must be identical in both the single-tenant and multitenant deployments.

• We recommend that you use a custom script or a third-party application like Postman to execute the API calls.

Migration Procedure

1. Export the single-tenant deployment and configuration data from a Cisco SD-WAN Manager instance controlling the overlay.

   Method	POST
   URL	https://ST-vManage-IP-address
   Endpoint	/dataservice/tenantmigration/export
   Authorization	Admin user credentials.
   Body	Required Format: Raw JSON { "desc": <tenant_description>, "name": <tenant_name>, "subdomain": <tenant_name>.<domain>, "orgName": <tenant_orgname > } Field Description: desc: A description of the tenant. The description can be up to 256 characters and can contain only alphanumeric characters. name: Unique name for the tenant in the multitenant deployment. subdomain: Fully qualified sub-domain name of the tenant. The sub-domain name must include the domain name of the service provider. For example, if managed-sp.com is the domain name of service provider, and the tenant name is Customer1, the tenant sub-domain name would be customer1.managed-sp.com. orgName: Name of the tenant organization. The organization name is case-sensitive.
   Response	Format: JSON { "processId": <vManage_process_ID>, }

   While exporting the data, Cisco SD-WAN Manager attempts to detach any CLI templates from the edge devices in preparation for the migration to the multitenant deployment. If prompted by Cisco SD-WAN Manager, detach CLI templates from the edge devices and execute the export API call again.

2. Check the status of the data export task in Cisco SD-WAN Manager. When the task succeeds, download the data using the URL https://ST-vManage-IP-address/dataservice/tenantmigration/download/default.tar.gz

3. On a multitenant Cisco SD-WAN Manager instance, import the data exported from the single-tenant overlay.

   Method	POST
   URL	https://MT-vManage-IP-address
   Endpoint	/dataservice/tenantmigration/import
   Authorization	Provider Admin user credentials.
   Body	Required Format: form-data Key Type: File Value: default.tar.gz
   Response	Format: JSON { "processId": <vManage_process_ID>, "migrationTokenURL": <token_URL>, }

   When the task succeeds, on the multitenant Cisco SD-WAN Manager, you can view the devices, templates, and policies imported from the single-tenant overlay.

4. Obtain the migration token using the token URL obtained in response to the API call in Step 3.

   Method	GET
   URL	https://MT-vManage-IP-address
   Endpoint	migrationTokenURL obtained in Step 3.
   Authorization	Provider Admin user credentials.
   Response	The migration token as a large blob of encoded text.

5. On the single-tenant Cisco SD-WAN Manager instance, initiate the migration of the overlay to the multitenant deployment.

   Method	POST
   URL	https://ST-vManage-IP-address
   Endpoint	dataservice/tenantmigration/networkMigration
   Authorization	Admin user credentials.
   Body	Required Format: Raw text Content: Migration token obtained in Step 4.
   Response	Format: JSON { "processId": <vManage_process_ID>, }

   In Cisco SD-WAN Manager, check the status of the migration task. As part of the migration task, the address of the multitenant Cisco SD-WAN Validator, and the service provider and tenant organization names are pushed to the WAN edge devices of the single-tenant overlay. If the task succeeds, WAN edge devices form control connections to controllers in the multitenant deployment; the WAN edge devices are no longer connected to the controllers of the single-tenant overlay.

   Attach any CLI templates detached from the edge devices (in Step 1) after migration to the multitenant deployment. Before you attach the templates, update the Cisco SD-WAN Validator IP address and the Organization name to match the configuration of the multitenant deployment.

   Note

   In the single-tenant deployment, if Cisco SD-WAN Manager-signed certificates are installed on cloud-based WAN edge devices, the certificates are cleared when the devices are migrated to the multitenant deployment. You must re-certify the devices on the multitenant Cisco SD-WAN Manager. If enterprise certificates are installed on the cloud-based WAN edge devices, the certificates are not affected by the migration. For more information, see Enterprise Certificates.

Before You Begin

• Manually migrate the serial number of the WAN edge device associated to a virtual account on the source Cisco SD-WAN Manager overlay in Cisco PNP to the destination virtual account.

• Ensure that you manually create the controller profile on the destination virtual account for on-prem to on-prem or cloud to on-prem deployments.

• Ensure that the source and destination Cisco SD-WAN Manager instances have the same Certificate Authority (CA). If not, recertify the devices after the migration is complete.

• Ensure that you check the CPU, memory, and disk size requirements of the destination overlay Cisco SD-WAN Controllers before the migration to meet the WAN edge forecast requirements.

• Ensure that there is no overlap between the configured system IP addresses of edge devices and the destination overlay controllers.

• Ensure that the destination single-tenant Cisco SD-WAN Manager does not have any configurations before migration. You can configure only mandatory admin settings and all other configurations can be done after data import.

• Ensure that the Cisco SD-WAN Control Components in the source and destination overlays are using the same software release. The migration process does not check for a software release mismatch and a mismatch blocks the import of tenant data, causing the migration to fail.

• Ensure that all devices in a tenant have connectivity to the Cisco SD-WAN Validator in the destination single-tenant overlay. The migration procedure supports a Cisco SD-WAN Validator on the single-tenant deployment configured either with IP or DNS.

  Push any required static route configuration to the devices before initiating any of the migration steps.

• Ensure that the WAN edge devices that are configured using CLI, device template, or configuration groups, have an IP host mapping to the Cisco SD-WAN Validator in the destination single-tenant overlay.

• Ensure that there are valid control connections from Cisco SD-WAN Manager to the WAN edge devices in the source overlay.

• Configure a maintenance window for the multitenant overlay before performing this procedure. See Configure or Cancel SD-WAN Manager Server Maintenance Window.

• We recommend that you use a custom script or a third-party application like Postman to execute the API calls.

Migration Procedure

1. Export the multitenant deployment configuration and statistical data from a Cisco SD-WAN Manager instance controlling the source overlay.

   Method	POST
   URL	https://MT-vManage-IP-address
   Endpoint	/dataservice/tenantmigration/export
   Authorization	Administrator user credentials.
   Body	Required Format: Raw JSON Example: { "name": "tenant1", "desc": "This is tenant1", "orgName": "vIPtela Inc MT to ST Migration Regression-Tenant1 Inc", "subDomain": "tenant1.mtreg.com", "wanEdgeForecast": 100, "migrationKey="tenant1TenantMigrationKey123", "isDestinationOverlayMT": false } Field descriptions: Note   Ensure that the name, desc, orgName, subdomain, and wanEdgeForecast match the tenant you wish to migrate. name: Unique name for the tenant in the multitenant deployment. The name should be between 8-32 characters and can contain only alphanumeric characters. desc: Description of the tenant. The description can be up to 256 characters and can contain only alphanumeric characters. orgName: Name of the tenant organization. The organization name is case-sensitive. subdomain: Fully qualified sub-domain name of the tenant. The sub-domain name must include the domain name of the service provider. For example, if managed-sp.com is the domain name of service provider, and the tenant name is Customer1, the tenant sub-domain name would be customer1.managed-sp.com. wanEdgeForecast: Number of WAN edge devices that the tenant can deploy. migrationKey: Migration key which is used to encrypt sensitive data during migration. The migration key should be between 8-32 characters and can contain only alphanumeric characters. isDestinationOverlayMT: Boolean variable which speficies if the migration is happening to a mutitenant overlay or not.
   Response	Format: JSON { "processId": <vManage_process_ID>, }

2. Check the status of the data export task in Cisco SD-WAN Manager. When the task is successfully complete, download the data from the following URL: https://MT-vManage-IP-address/dataservice/tenantmigration/download/default.tar.gz

3. Import the data to the single-tenant instance, as follows:

   1. Execute the following API:

      Method	POST
      URL	https://ST-vManage-IP-address
      Endpoint	/dataservice/tenantmigration/import/{migrationKey} Use the same migration key specified earlier.
      Authorization	Provider administrator user credentials.
      Body	Required Format: form-data Key Type: File Value: default.tar.gz
      Response	Format: JSON { "processId": <vManage_process_ID>, "migrationTokenURL": <token_URL>, }

   2. When the task is complete, on the single-tenant Cisco SD-WAN Manager, you can view the devices, templates, and policies imported from the multitenant overlay.

4. After the import, update information related to the device templates, policies, and other deployment-specific parameters. Check and update the administrator settings as some of the administrator settings specific to the source overlay are not exported. The import does not override the administrator settings that are already configured in destination Cisco SD-WAN Manager.

5. If a centralized policy is present on the source tenant, the migration copies the policy to the destination overlay. We recommend creating Cisco SD-WAN Controller templates and attaching them to the devices. Apply the centralized policy to devices in the destination overlay before proceeding.

6. Obtain the migration token using the token URL from the previous step.

   Method	GET
   URL	https://ST-vManage-IP-address
   Endpoint	migrationTokenURL obtained in the previous step.
   Authorization	Provider administrator user credentials.
   Response	The migration token as a large encoded text.

7. On the multitenant Cisco SD-WAN Manager instance, initiate the migration of the overlay to the single-tenant deployment.

   Method	POST
   URL	https://MT-vManage-IP-address
   Endpoint	dataservice/tenantmigration/networkMigration
   Authorization	Administrator user credentials.
   Body	Required Format: Raw text Content: Migration token obtained in the previous step.
   Response	Format: JSON { "processId": <vManage_process_ID>, }

   In Cisco SD-WAN Manager, check the status of the migration task. When the task succeeds, WAN edge devices form control connections to controllers in the single-tenant deployment; the WAN edge devices are no longer connected to the controllers of the multitenant overlay.

8. After the migration is successfully complete, perform the following tasks:

   • If WAN edge devices have Cisco SD-WAN Manager signed certificates in the source setup, the certificates are cleared from the device during migration and control connections are lost. Recertify the devices in the destination.

   • The passwords are updated to the default password in the destination overlay for users created on a tenant in the source overlay. Make any configuration changes specific to the destination overlay.

   • Delete the tenant on the source overlay after migration and verification is complete.

Prerequisites

Minimum software requirements for Cisco Catalyst SD-WAN controllers and WAN edge devices in the multitenant overlay to be migrated:

Restrictions

• This migration procedure applies only to Cisco Catalyst SD-WAN controllers deployed on premises.

• The multitenant overlay can only be migrated to a setup in which Cisco SD-WAN Manager instances run Cisco vManage Release 20.6.1 software and Cisco Catalyst SD-WAN controllers run Cisco SD-WAN Release 20.6.1 software.

• This migration procedure cannot be used to merge two or more multitenant overlays. Only one multitenant overlay can be migrated to the new setup at a time.

Migration Procedure

1. Upgrade the software on the three Cisco SD-WAN Manager instances in the cluster to Cisco vManage Release 20.6.1. For more information, see Upgrade Cisco SD-WAN Manager Cluster.

   Note	Run the command request nms configuration-db upgrade on only one of the Cisco SD-WAN Manager instances.

2. After the Cisco SD-WAN Manager software is upgraded to Cisco vManage Release 20.6.1, log in to the Cisco SD-WAN Manager.

   You're prompted to set a new password.

   1. Enter a new password that adheres to the password guidelines.

3. Upload the Cisco SD-WAN Release 20.6.1 software to Cisco SD-WAN Manager. For more information, see Add an Image to the Software Repository.

4. Upgrade the Cisco SD-WAN Validator software to Cisco SD-WAN Release 20.6.1. For more information, see Upgrade the Software Image on a Device.

5. Create two Cisco SD-WAN Controller instances running Cisco SD-WAN Release 20.6.1 software. See Deploy the Cisco SD-WAN Controller.

   Note	With two Cisco SD-WAN Controller instances, you can support up to 24 tenants. To support up to 50 tenants, create six Cisco SD-WAN Controller instances.

   1. Add Cisco SD-WAN Controllers to the overlay network.

   The Provider Dashboard shows the new Cisco SD-WAN Controllers running Cisco SD-WAN Release 20.6.1 software. The Tenant Dashboard shows the older Cisco SD-WAN Controllers running Cisco SD-WAN Release 20.3.3 software.

6. Enable the maintenance window on Cisco SD-WAN Manager. For more information, see Configure or Cancel SD-WAN Manager Server Maintenance Window.

   A maintenance window of 3 to 4 hours is recommended.

7. Migrate the tenant configuration from the older tenant-specific Cisco SD-WAN Controllers running Cisco SD-WAN Release 20.3.3 software to the new shared Cisco SD-WAN Controllers running Cisco SD-WAN Release 20.6.1 software.

   Method	POST
   URL	https://<vmanageip>:<port>
   Endpoint	dataservice/tenant/vsmart-mt/migrate
   Authorization	Provider admin user credentials.
   Body	Required Format: Raw JSON {}
   Response	Format: JSON { "processId": <vManage_process_ID>, }

   In Cisco SD-WAN Manager, check the status of the migration task using the processId from the API response. During the migration task, the following changes are affected:

   1. The older Cisco SD-WAN Controllers are invalidated and deleted from the overlay network.

   2. In the tenant view, the older Cisco SD-WAN Controllers are removed from the Tenant Dashboard, and the Devices and the Certificates page.

   3. The tenant WAN edge devices are connected to the new Cisco SD-WAN Controllers.

8. (Optional) Upgrade the Cisco IOS XE Catalyst SD-WAN device software to Cisco IOS XE Catalyst SD-WAN Release 17.6.1a. For more information, see Upgrade the Software Image on a Device and Activate a New Software Image.

   Tip	It is not necessary to upgrade the tenant WAN edge device software in the same maintenance window in which you migrate the multitenant overlay. However, we recommend that you upgrade the tenant WAN edge device software within a few weeks of the migration.

Verify the Migration

1. In the provider view, perform the following checks:

   1. From the Main Dashboard page, verify whether the tenant WAN edge devices are connected to the new multitenant Cisco SD-WAN Controllers.

   2. View Tenants Associated with a Cisco SD-WAN Controller.

   3. On the Cisco SD-WAN Controller CLI, run the command show control connections . In the command output, verify that control connections are established between the Cisco SD-WAN Controller and the tenant WAN edge devices.

2. In the provider-as-tenant view, verify whether the multitenant Cisco SD-WAN Controllers appear on the Tenant Dashboard.

Prerequisites

Minimum software requirements for Cisco Catalyst SD-WAN controllers and WAN edge devices:

Upgrade Procedure

1. Upgrade the software on the three Cisco SD-WAN Manager instances in the cluster to Cisco vManage Release 20.6.1 or a later release. For more information, see Upgrade Cisco SD-WAN Manager Cluster.

   Note	Skip the step to upgrade the configuration-db service using the command request nms configuration-db upgrade .

2. After the Cisco SD-WAN Manager software is upgraded to Cisco vManage Release 20.6.1 or a later release, log in to the Cisco SD-WAN Manager.

3. Upload the Cisco SD-WAN Release 20.6.1 or a later release and the Cisco IOS XE Catalyst SD-WAN Release 17.6.1a or a later release software to Cisco SD-WAN Manager. For more information, see Add an Image to the Software Repository.

4. Upgrade the Cisco SD-WAN Validator software to Cisco SD-WAN Release 20.6.1 or a later release. For more information, see Upgrade the Software Image on a Device and Activate a New Software Image.

5. Enable maintenance window on Cisco SD-WAN Manager. For more information, see Configure or Cancel SD-WAN Manager Server Maintenance Window.

6. Upgrade the Cisco Catalyst SD-WAN Controller software to Cisco SD-WAN Release 20.6.1 or a later release. For more information, see Upgrade the Software Image on a Device and Activate a New Software Image.

7. Upgrade the Cisco IOS XE Catalyst SD-WAN device software to Cisco IOS XE Catalyst SD-WAN Release 17.6.1a or a later release. For more information, see Upgrade the Software Image on a Device and Activate a New Software Image.

   Tip	We recommend that you upgrade the WAN edge device software in the same maintenance window. If the WAN edge device software is not upgraded within the OMP graceful restart window, traffic may be lost.

If a Multitenant Cisco SD-WAN Manager cluster or the data center hosting the Cisco SD-WAN Manager nodes in the cluster fail, you can recover from the failure by activating a standby Cisco SD-WAN Manager cluster. You can perform disaster recovery as follows:

1. Deploy and configure a standby Cisco SD-WAN Manager cluster.

   The standby Cisco SD-WAN Manager cluster is not part of the overlay network and is not active.

2. Back up the configuration database of the active Cisco SD-WAN Manager cluster periodically.

   Choose a Cisco SD-WAN Manager node in the cluster that hosts the configuration database service and back up the configuration database.

3. If the active Cisco SD-WAN Manager cluster fails, restore the most recent configuration database on the standby Cisco SD-WAN Manager cluster, activate the standby Cisco SD-WAN Manager cluster, and remove the previously active Cisco SD-WAN Manager cluster from the overlay network.

   Choose a Cisco SD-WAN Manager node in the cluster that hosts the configuration database service and restore the configuration database backed up from the previously active Cisco SD-WAN Manager cluster.

To test disaster recovery, you can simulate a scenario in which the active Cisco SD-WAN Manager cluster fails. One way to simulate such a failure would be by disabling the tunnel interface as described in this document.

Prerequisites

• The number of Cisco SD-WAN Manager nodes in the active and standby clusters must be identical.

• Each Cisco SD-WAN Manager node in the active and standby clusters must run the same Cisco SD-WAN Manager software release.

• Each Cisco SD-WAN Manager node in the active and standby clusters must be able to connect to the WAN transport IP address of the Cisco SD-WAN Validator in the overlay network.

• Initially, the tunnel interfaces of the Cisco SD-WAN Manager nodes in the standby cluster must be disabled.

• The Cisco SD-WAN Manager nodes in the standby cluster must be certified.

• The clock of every Cisco SD-WAN Manager node in the standby cluster must be synchronized with the clocks of the Cisco Catalyst SD-WAN controllers and WAN edge devices in the overlay network. If NTP is configured on the overlay, configure the same on the standby Cisco SD-WAN Manager nodes.

• The Cisco SD-WAN Manager nodes in the active and standby clusters should use identical neo4j credentials.

Restrictions

• Do not interrupt any active processes while backing up the configuration database.

• If you wish to enable SD-AVC, you must do so before the restoring the configuration database on standby Cisco SD-WAN Manager node.

Configure a Standby Cisco SD-WAN Manager Cluster

1. Configure the standby Cisco SD-WAN Manager nodes with a similar running configuration as the active Cisco SD-WAN Manager nodes. Install local certificates on the standby Cisco SD-WAN Manager nodes.

   Note	The running configuration on a standby Cisco SD-WAN Manager is usually identical to that of an active Cisco SD-WAN Manager node. However, you must ensure that settings such as the system IP address and the tunnel interface IP address are unique.

2. On the standby Cisco SD-WAN Manager nodes, shut down the transport interface in VPN 0: On the CLI, include the shutdown command in the transport interface configuration.

3. Create a standby cluster using the standby Cisco SD-WAN Manager nodes.

With the standby Cisco SD-WAN Manager nodes configured in this manner, the overlay network is not aware of the standby Cisco SD-WAN Manager cluster.

Back Up the Active Cisco SD-WAN Manager Cluster Configuration

Back up the full configuration database of the active Cisco vManange cluster periodically. Additionally, take snapshots of the active Cisco SD-WAN Manager virtual machines.

1. Choose an active Cisco SD-WAN Manager node that hosts the configuration database service and export a backup the configuration database. On the CLI of the Cisco SD-WAN Manager node, run the following command: request nms configuration-db backup path file-path

   The command backs up the configuration database in a .tar.gz file and saves the file in the specified file-path .

   In the following example, the database is backed up to a file named db_backup.tar.gz in the /home/admin/ directory.

   Active-vManage# request nms configuration-db backup path /home/admin/db_backup
   Successfully saved database to /home/admin/db_backup.tar.gz

2. Choose a standby Cisco SD-WAN Manager node that hosts the configuration database service and copy the configuration database backup to this node.

   In the following example, db_backup.tar.gz is copied from the active Cisco SD-WAN Manager node to the /home/admin/ directory of a standby Cisco SD-WAN Manager node.

   Active-vManage# request execute vpn 512 scp /home/admin/db_backup.tar.gz admin@10.126.93.92:/home/admin
   The authenticity of host '10.126.93.92 (10.126.93.92)' can't be established.
   ECDSA key fingerprint is SHA256:jTjJWQ0UNHvlrBUxWzNjd8mUz819gPf51MeopsgDlAc.
   Are you sure you want to continue connecting (yes/no)? yes
   Warning: Permanently added '10.126.93.92' (ECDSA) to the list of known hosts.
   viptela 18.4.5 
    
   admin@10.126.93.92's password: 
   db_backup.tar.gz                                100%  399KB   4.4MB/s   00:00

Restore Cisco SD-WAN Manager Cluster Using the Configuration Database Backup

Restore the most recent backup of the configuration database from the active Cisco SD-WAN Manager cluster on the standby Cisco SD-WAN Manager node to which you copied this backup.

Note

The restore operation does not restore all the information included in the configuration database. Cisco SD-WAN Manager configurations such as users and repositories must be configured on the standby Cisco SD-WAN Manager node after the configuration database is restored using the backup. When you complete the steps that follow, the previously active Cisco SD-WAN Manager nodes cannot be reused. To reuse the nodes, you must perform additional steps that are beyond the scope of this document.

1. On the CLI of the standby Cisco SD-WAN Manager node, run the following command: request nms configuration-db restore path file-path

   In the following example, the configuration database is restored using the backup file db_backup.tar.gz.

   Standby-vManage# request nms configuration-db restore path /home/admin/db_backup.tar.gz
   Configuration database is running in a standalone mode
   Importing database...Successfully restored database

2. Verify that the appropriate services are running on the standby Cisco SD-WAN Manager nodes: On the CLI of each standby Cisco SD-WAN Manager node, run the request nms all status command. From the command output, verify the services running on the node.

3. Verify that every standby Cisco SD-WAN Manager node has a list of all the active and standby Cisco SD-WAN Manager nodes.

   1. From the Cisco SD-WAN Manager menu, choose Configuration > Devices > Controllers.

      Note	Starting from Cisco IOS XE Catalyst SD-WAN Release 17.13.1a, the Controllers tab is renamed as the Control Components tab to stay consistent with Cisco Catalyst SD-WAN rebranding.

   2. Verify that the page displays all active and standby Cisco SD-WAN Manager nodes.

4. On the standby Cisco SD-WAN Manager nodes, enable the transport interface on VPN 0.

   Use one of the following two methods:

   1. Enable the transport interface in VPN 0: On the CLI of each standby Cisco SD-WAN Manager node, run the no shutdown command.

      Active-vManage# config
      Active-vManage(config)# vpn 0 interface interface-name
      Active-vManage(config-interface)# no shutdown
      Active-vManage(config-interface)# commit and-quit

   2. Activate the tunnel interface in VPN 0: On the CLI of each standby Cisco SD-WAN Manager node, run the tunnel-interface command.

      Active-vManage# config
      Active-vManage(config)# vpn 0 interface interface-name
      Active-vManage(config-interface)# tunnel-interface
      Active-vManage(config-interface)# commit and-quit

5. Add each standby Cisco SD-WAN Manager node to the overlay network.

   1. From the Cisco SD-WAN Manager menu, choose Configuration > Devices.

   2. Click Controllers.

      Note	Starting from Cisco IOS XE Catalyst SD-WAN Release 17.13.1a, the Controllers tab is renamed as the Control Components tab to stay consistent with Cisco Catalyst SD-WAN rebranding.

   3. For a Cisco SD-WAN Validator, click ... and click Edit.

   4. In the Edit dialog box, enter the following details of the Cisco SD-WAN Validator: WAN transport IP address, username, and password.

   5. Repeat Step 5c and Step 5d for every Cisco SD-WAN Validator.

6. Disconnect the active Cisco SD-WAN Manager nodes from the overlay network.

   Note	In a lab environment, where you are simulating a disaster scenario, you can perform this step. However, if you cannot reach Cisco SD-WAN Manager instances in an actual disaster scenario, you may not be able to perform this step and can omit the step.

   Use one of the following two methods:

   1. Shut down the transport interface in VPN 0: On the CLI of each active Cisco SD-WAN Manager node, run the shutdown command.

      Active-vManage# config
      Active-vManage(config)# vpn 0 interface interface-name
      Active-vManage(config-interface)# shutdown
      Active-vManage(config-interface)# commit and-quit

   2. Deactivate the tunnel interface in VPN 0: On the CLI of each active Cisco SD-WAN Manager node, run the no tunnel-interface command.

      Active-vManage# config
      Active-vManage(config)# vpn 0 interface interface-name
      Active-vManage(config-interface)# no tunnel-interface
      Active-vManage(config-interface)# commit and-quit

7. From the standby Cisco SD-WAN Manager, send the updated controller and device list to the Cisco SD-WAN Validator.

   Send the list of controllers:

   1. From the Cisco SD-WAN Manager menu, choose Configuration > Certificates.

   2. Click Controllers.

      Note	Starting from Cisco IOS XE Catalyst SD-WAN Release 17.13.1a, the Controllers tab is renamed as the Control Components tab to stay consistent with Cisco Catalyst SD-WAN rebranding.

   3. Click Send to Validator.

      Wait for the configuration task to complete. When the task is complete,

      • The standby Cisco SD-WAN Manager nodes become the active Cisco SD-WAN Manager nodes.

      • The previously active Cisco SD-WAN Manager nodes are no longer part of the overlay network.

      • The active Cisco SD-WAN Manager nodes have the configuration from the most recent configuration database backup.

      • Every controller establishes connection with the other controllers in the network.

   4. Click WAN Edge List.

   5. Click Send to Controllers.

8. Verify that the following are intact:

   • Policies

   • Templates

   • Controller and WAN edge device lists

9. Verify the valid Cisco SD-WAN Manager nodes.

   1. Log in to the CLI of each Cisco SD-WAN Validator and run the show orchestrator valid-vmanage-id command.

      In the command output, verify that the chassis numbers of the active and previously active Cisco SD-WAN Manager nodes are listed.

   2. Log in to the CLI of a WAN edge device and run the show control valid-vmanage-id command.

      In the command output, verify that the chassis numbers of the active and previously active Cisco SD-WAN Manager nodes are listed. Also, check whether the device is connected to the active Cisco SD-WAN Manager nodes and the Cisco Catalyst SD-WAN Controller.

10. Invalidate the previously active Cisco SD-WAN Manager nodes.

    Note	After you invalidate the Cisco SD-WAN Manager nodes, the nodes cannot be reused without performing additional steps that are beyond the scope of this document.

    1. From the Cisco SD-WAN Manager menu, choose Configuration > Certificates.

    2. Click Controllers.

       Note	Starting from Cisco IOS XE Catalyst SD-WAN Release 17.13.1a, the Controllers tab is renamed as the Control Components tab to stay consistent with Cisco Catalyst SD-WAN rebranding.

    3. For each previously active Cisco SD-WAN Manager node, click ... and click Invalidate.

11. Verify the valid Cisco SD-WAN Manager nodes.

    1. Log in to the CLI of each Cisco SD-WAN Validator and run the show orchestrator valid-vmanage-id command.

       In the command output, verify that the chassis numbers of only the active Cisco SD-WAN Manager nodes are listed.

    2. Log in to the CLI of WAN edge device and run the show control valid-vmanage-id command.

       In the command output, verify that the chassis numbers of only the active Cisco SD-WAN Manager nodes are listed. Also, check whether the device is connected to the active Cisco SD-WAN Manager nodes and the Cisco Catalyst SD-WAN Controller.

The Cisco SD-WAN Manager cluster that was initially the standby cluster is now the active Cisco SD-WAN Manager cluster.

If a Multitenant Cisco SD-WAN Manager cluster or the data center hosting the Cisco SD-WAN Manager nodes in the cluster fail, you can recover from the failure by activating a standby Cisco SD-WAN Manager cluster. You can perform disaster recovery as follows:

1. Deploy and configure a standby Cisco SD-WAN Manager cluster.

   The standby Cisco SD-WAN Manager cluster is not part of the overlay network and is not active.

2. Back up the configuration database of the active Cisco SD-WAN Manager cluster periodically.

   Choose a Cisco SD-WAN Manager node in the cluster that hosts the configuration database service and back up the configuration database.

3. If the active Cisco SD-WAN Manager cluster fails, restore the most recent configuration database on the standby Cisco SD-WAN Manager cluster, activate the standby Cisco SD-WAN Manager cluster, and remove the previously active Cisco SD-WAN Manager cluster from the overlay network.

   Choose a Cisco SD-WAN Manager node in the cluster that hosts the configuration database service and restore the configuration database backed up from the previously active Cisco SD-WAN Manager cluster.

To test disaster recovery, you can simulate a scenario in which the active Cisco SD-WAN Manager cluster fails. One way to simulate such a failure would be by disabling the tunnel interface as described in this document.

The following disaster recovery procedure applies to an overlay network in which Cisco vEdge Cloud routers are deployed at branch locations.

Prerequisites

• The number of Cisco SD-WAN Manager nodes in the active and standby clusters must be identical.

• Each Cisco SD-WAN Manager node in the active and standby clusters must run the same Cisco SD-WAN Manager software release.

• Each Cisco SD-WAN Manager node in the active and standby clusters must be able to connect to the WAN transport IP address of the Cisco SD-WAN Validator in the overlay network.

• Initially, the tunnel interfaces of the Cisco SD-WAN Manager nodes in the standby cluster must be disabled.

• The Cisco SD-WAN Manager nodes in the standby cluster must be certified.

• The clock of every Cisco SD-WAN Manager node in the standby cluster must be synchronized with the clocks of the Cisco Catalyst SD-WAN controllers and WAN edge devices in the overlay network. If NTP is configured on the overlay, configure the same on the standby Cisco SD-WAN Manager nodes.

• The Cisco SD-WAN Manager nodes in the active and standby clusters should use identical neo4j credentials.

Restrictions

• Do not interrupt any active processes while backing up the configuration database.

• If you wish to enable SD-AVC, you must do so before the restoring the configuration database on standby Cisco SD-WAN Manager node.

Configure a Standby Cisco SD-WAN Manager Cluster

1. Configure the standby Cisco SD-WAN Manager nodes with a similar running configuration as the active Cisco SD-WAN Manager nodes. Install local certificates on the standby Cisco SD-WAN Manager nodes.

   Note	The running configuration on a standby Cisco SD-WAN Manager is usually identical to that of an active Cisco SD-WAN Manager node. However, you must ensure that settings such as the system IP address and the tunnel interface IP address are unique.

2. On the standby Cisco SD-WAN Manager nodes, shut down the transport interface in VPN 0: On the CLI, include the shutdown command in the transport interface configuration.

3. Create a standby cluster using the standby Cisco SD-WAN Manager nodes.

With the standby Cisco SD-WAN Manager nodes configured in this manner, the overlay network is not aware of the standby Cisco SD-WAN Manager cluster.

Back Up the Active Cisco SD-WAN Manager Cluster Configuration

Back up the full configuration database of the active Cisco vManange cluster periodically. Additionally, take snapshots of the active Cisco SD-WAN Manager virtual machines.

1. Choose an active Cisco SD-WAN Manager node that hosts the configuration database service and export a backup the configuration database. On the CLI of the Cisco SD-WAN Manager node, run the following command: request nms configuration-db backup path file-path

   The command backs up the configuration database in a .tar.gz file and saves the file in the specified file-path .

   In the following example, the database is backed up to a file named db_backup.tar.gz in the /home/admin/ directory.

   Active-vManage# request nms configuration-db backup path /home/admin/db_backup
   Successfully saved database to /home/admin/db_backup.tar.gz

2. Choose a standby Cisco SD-WAN Manager node that hosts the configuration database service and copy the configuration database backup to this node.

   In the following example, db_backup.tar.gz is copied from the active Cisco SD-WAN Manager node to the /home/admin/ directory of a standby Cisco SD-WAN Manager node.

   Active-vManage# request execute vpn 512 scp /home/admin/db_backup.tar.gz admin@10.126.93.92:/home/admin
   The authenticity of host '10.126.93.92 (10.126.93.92)' can't be established.
   ECDSA key fingerprint is SHA256:jTjJWQ0UNHvlrBUxWzNjd8mUz819gPf51MeopsgDlAc.
   Are you sure you want to continue connecting (yes/no)? yes
   Warning: Permanently added '10.126.93.92' (ECDSA) to the list of known hosts.
   viptela 18.4.5 
    
   admin@10.126.93.92's password: 
   db_backup.tar.gz                                100%  399KB   4.4MB/s   00:00

Restore Cisco SD-WAN Manager Cluster Using the Configuration Database Backup

Restore the most recent backup of the configuration database from the active Cisco SD-WAN Manager cluster on the standby Cisco SD-WAN Manager node to which you copied this backup.

Note

The restore operation does not restore all the information included in the configuration database. Cisco SD-WAN Manager configurations such as users and repositories must be configured on the standby Cisco SD-WAN Manager node after the configuration database is restored using the backup. When you complete the steps that follow, the previously active Cisco SD-WAN Manager nodes cannot be reused. To reuse the nodes, you must perform additional steps that are beyond the scope of this document.

1. On the CLI of the standby Cisco SD-WAN Manager node, run the following command: request nms configuration-db restore path file-path

   In the following example, the configuration database is restored using the backup file db_backup.tar.gz.

   Standby-vManage# request nms configuration-db restore path /home/admin/db_backup.tar.gz
   Configuration database is running in a standalone mode
   Importing database...Successfully restored database

2. Verify that the appropriate services are running on the standby Cisco SD-WAN Manager nodes: On the CLI of each standby Cisco SD-WAN Manager node, run the request nms all status command. From the command output, verify the services running on the node.

3. Verify that every standby Cisco SD-WAN Manager node has a list of all the active and standby Cisco SD-WAN Manager nodes.

   1. From the Cisco SD-WAN Manager menu, choose Configuration > Devices > Controllers.

      Note	Starting from Cisco IOS XE Catalyst SD-WAN Release 17.13.1a, the Controllers tab is renamed as the Control Components tab to stay consistent with Cisco Catalyst SD-WAN rebranding.

   2. Verify that the page displays all active and standby Cisco SD-WAN Manager nodes.

4. Log in to the CLI of each Cisco SD-WAN Validator and run the show orchestrator valid-vmanage-id command.

   In the command output, verify that the chassis numbers of the active and previously active Cisco SD-WAN Manager nodes are listed.

5. Log in to the CLI of Cisco vEdge Cloud router and run the show control valid-vmanage-id command.

   In the command output, verify that the chassis numbers of the active and previously active Cisco SD-WAN Manager nodes are listed.

6. On the standby Cisco SD-WAN Manager nodes, enable the transport interface on VPN 0.

   Use one of the following two methods:

   1. Enable the transport interface in VPN 0: On the CLI of each standby Cisco SD-WAN Manager node, run the no shutdown command.

      Active-vManage# config
      Active-vManage(config)# vpn 0 interface interface-name
      Active-vManage(config-interface)# no shutdown
      Active-vManage(config-interface)# commit and-quit

   2. Activate the tunnel interface in VPN 0: On the CLI of each standby Cisco SD-WAN Manager node, run the tunnel-interface command.

      Active-vManage# config
      Active-vManage(config)# vpn 0 interface interface-name
      Active-vManage(config-interface)# tunnel-interface
      Active-vManage(config-interface)# commit and-quit

7. Add each standby Cisco SD-WAN Manager node to the overlay network.

   1. From the Cisco SD-WAN Manager menu, choose Configuration > Devices.

   2. Click Controllers.

      Note	Starting from Cisco IOS XE Catalyst SD-WAN Release 17.13.1a, the Controllers tab is renamed as the Control Components tab to stay consistent with Cisco Catalyst SD-WAN rebranding.

   3. For a Cisco SD-WAN Validator, click ... and click Edit.

   4. In the Edit dialog box, enter the following details of the Cisco SD-WAN Validator: WAN transport IP address, username, and password.

   5. Repeat Step 7c and Step 7d for every Cisco SD-WAN Validator.

8. Disconnect the active Cisco SD-WAN Manager nodes from the overlay network.

   Note	In a lab environment, where you are simulating a disaster scenario, you can perform this step. However, if you cannot reach Cisco SD-WAN Manager instances in an actual disaster scenario, you may not be able to perform this step and can omit this step.

   Use one of the following two methods:

   1. Shut down the transport interface in VPN 0: On the CLI of each active Cisco SD-WAN Manager node, run the shutdown command.

      Active-vManage# config
      Active-vManage(config)# vpn 0 interface interface-name
      Active-vManage(config-interface)# shutdown
      Active-vManage(config-interface)# commit and-quit

   2. Deactivate the tunnel interface in VPN 0: On the CLI of each active Cisco SD-WAN Manager node, run the no tunnel-interface command.

      Active-vManage# config
      Active-vManage(config)# vpn 0 interface interface-name
      Active-vManage(config-interface)# no tunnel-interface
      Active-vManage(config-interface)# commit and-quit

9. From the standby Cisco SD-WAN Manager, send the updated controller and device list to the Cisco SD-WAN Validator.

   Send the list of controllers:

   1. From the Cisco SD-WAN Manager menu, choose Configuration > Certificates.

   2. Click Controllers.

      Note	Starting from Cisco IOS XE Catalyst SD-WAN Release 17.13.1a, the Controllers tab is renamed as the Control Components tab to stay consistent with Cisco Catalyst SD-WAN rebranding.

   3. Click Send to Validator.

      Wait for the configuration task to complete. When the task is complete,

      • The standby Cisco SD-WAN Manager nodes become the active Cisco SD-WAN Manager nodes.

      • The previously active Cisco SD-WAN Manager nodes are no longer part of the overlay network.

      • The active Cisco SD-WAN Manager nodes have the configuration from the most recent configuration database backup.

      • Every controller establishes connection with the other controllers in the network.

   4. Click WAN Edge List.

   5. Click Send to Controllers.

10. Verify that the following are intact:

    • Policies

    • Templates

    • Controller and WAN edge device lists

11. Verify the valid Cisco SD-WAN Manager nodes.

    1. Log in to the CLI of each Cisco SD-WAN Validator and run the show orchestrator valid-vmanage-id command.

       In the command output, verify that the chassis numbers of the active and previously active Cisco SD-WAN Manager nodes are listed.

    2. Log in to the CLI of a Cisco vEdge Cloud router and run the show control valid-vmanage-id command.

       In the command output, verify that the chassis numbers of the active and previously active Cisco SD-WAN Manager nodes are listed. Also, check whether the device is connected to the active Cisco SD-WAN Manager nodes and the Cisco Catalyst SD-WAN Controller.

12. Invalidate the previously active Cisco SD-WAN Manager nodes.

    The previously active Cisco SD-WAN Manager is the certificate issuer for the cloud WAN edge devices. The active Cisco SD-WAN Manager issues certificates to the cloud WAN edge devices only after the previously active Cisco SD-WAN Manager nodes are invalidated.

    Note

    After you invalidate the Cisco SD-WAN Manager nodes, the nodes cannot be reused without performing additional steps that are beyond the scope of this document. When you invalidate the previously active Cisco SD-WAN Manager nodes, Cisco SD-WAN Manager marks the nodes as invalid and sends an update to all controllers. However, Cisco SD-WAN Manager does not send an updated list of valid Cisco SD-WAN Manager UUIDs to Cisco SD-WAN Validator immediately because the previously active Cisco SD-WAN Manager is the CA for the cloud WAN edge devices. So, the output of the show orchestrator valid-vmanage-id command on a Cisco SD-WAN Validator includes the UUIDs of the invalidated Cisco SD-WAN Manager nodes. Cisco SD-WAN Manager has a scheduled task that runs every 24 hours and checks to see if all the cloud WAN edges have been moved to the active Cisco SD-WAN Manager. Cisco SD-WAN Manager sends the updated list of valid Cisco SD-WAN Manager UUIDs to Cisco SD-WAN Validator only after the cloud WAN edge devices have been moved to the active Cisco SD-WAN Manager. After this list is received, the output of the show orchestrator valid-vmanage-id command on a Cisco SD-WAN Validator does not include the UUIDs of the invalidated Cisco SD-WAN Manager nodes.

    1. From the Cisco SD-WAN Manager menu, choose Configuration > Certificates.

    2. Click Controllers.

       Note	Starting from Cisco IOS XE Catalyst SD-WAN Release 17.13.1a, the Controllers tab is renamed as the Control Components tab to stay consistent with Cisco Catalyst SD-WAN rebranding.

    3. For each previously active Cisco SD-WAN Manager node, click ... and click Invalidate.

13. Verify the valid Cisco SD-WAN Manager nodes after 24 hours.

    1. Log in to the CLI of each Cisco SD-WAN Validator and run the show orchestrator valid-vmanage-id command.

       In the command output, verify that the chassis numbers of only the active Cisco SD-WAN Manager nodes are listed.

    2. Log in to the CLI of WAN edge device and run the show control valid-vmanage-id command.

       In the command output, verify that the chassis numbers of only the active Cisco SD-WAN Manager nodes are listed. Also, check whether the device is connected to the active Cisco SD-WAN Manager nodes and the Cisco Catalyst SD-WAN Controllers.

The Cisco SD-WAN Manager cluster that was initially the standby cluster is now the active Cisco SD-WAN Manager cluster.

If a Multitenant Cisco SD-WAN Manager cluster or the data center hosting the Cisco SD-WAN Manager nodes in the cluster fail, you can recover from the failure by activating a standby Cisco SD-WAN Manager cluster. You can perform disaster recovery as follows:

1. Deploy and configure a standby Cisco SD-WAN Manager cluster.

   The standby Cisco SD-WAN Manager cluster is not part of the overlay network and is not active.

2. Back up the configuration database of the active Cisco SD-WAN Manager cluster periodically.

   Choose a Cisco SD-WAN Manager node in the cluster that hosts the configuration database service and back up the configuration database.

3. If the active Cisco SD-WAN Manager cluster fails, restore the most recent configuration database on the standby Cisco SD-WAN Manager cluster, activate the standby Cisco SD-WAN Manager cluster, and remove the previously active Cisco SD-WAN Manager cluster from the overlay network.

   Choose a Cisco SD-WAN Manager node in the cluster that hosts the configuration database service and restore the configuration database backed up from the previously active Cisco SD-WAN Manager cluster.

To test disaster recovery, you can simulate a scenario in which the active Cisco SD-WAN Manager cluster fails. One way to simulate such a failure would be by disabling the tunnel interface as described in this document.

The following procedure applies to a scenario in which an initially active Cisco SD-WAN Manager cluster or the data center hosting the cluster failed and the standby Cisco SD-WAN Manager cluster was configured to be the active Cisco SD-WAN Manager cluster. If the cluster that was initially active becomes operational again, it serves as a standby cluster. By completing the following procedure, you can turn this standby cluster into the active cluster.

Check the Configuration of the Standby Cisco SD-WAN Manager

1. Check whether the running configuration of the standby Cisco SD-WAN Manager nodes is similar to the running configuration of the active Cisco SD-WAN Manager nodes. Local certificates must be installed on the standby Cisco SD-WAN Manager nodes.

   Note	The running configuration on a standby Cisco SD-WAN Manager is usually identical to that of an active Cisco SD-WAN Manager node. However, you must ensure that settings such as the system IP address and the tunnel interface IP address are unique.

2. On the standby Cisco SD-WAN Manager nodes, shut down the transport interface in VPN 0: On the CLI, include the shutdown command in the transport interface configuration.

With the standby Cisco SD-WAN Manager nodes configured in this manner, the overlay network is not aware of the standby Cisco SD-WAN Manager cluster.

Back Up the Active Cisco SD-WAN Manager Cluster Configuration

Back up the full configuration database of the active Cisco vManange cluster periodically. Additionally, take snapshots of the active Cisco SD-WAN Manager virtual machines.

1. Choose an active Cisco SD-WAN Manager node that hosts the configuration database service and export a backup the configuration database. On the CLI of the Cisco SD-WAN Manager node, run the following command: request nms configuration-db backup path file-path

   The command backs up the configuration database in a .tar.gz file and saves the file in the specified file-path .

   In the following example, the database is backed up to a file named db_backup.tar.gz in the /home/admin/ directory.

   Active-vManage# request nms configuration-db backup path /home/admin/db_backup
   Successfully saved database to /home/admin/db_backup.tar.gz

2. Choose a standby Cisco SD-WAN Manager node that hosts the configuration database service and copy the configuration database backup to this node.

   In the following example, db_backup.tar.gz is copied from the active Cisco SD-WAN Manager node to the /home/admin/ directory of a standby Cisco SD-WAN Manager node.

   Active-vManage# request execute vpn 512 scp /home/admin/db_backup.tar.gz admin@10.126.93.92:/home/admin
   The authenticity of host '10.126.93.92 (10.126.93.92)' can't be established.
   ECDSA key fingerprint is SHA256:jTjJWQ0UNHvlrBUxWzNjd8mUz819gPf51MeopsgDlAc.
   Are you sure you want to continue connecting (yes/no)? yes
   Warning: Permanently added '10.126.93.92' (ECDSA) to the list of known hosts.
   viptela 18.4.5 
    
   admin@10.126.93.92's password: 
   db_backup.tar.gz                                100%  399KB   4.4MB/s   00:00

Restore Cisco SD-WAN Manager Cluster Using the Configuration Database Backup

Restore the most recent backup of the configuration database from the active Cisco SD-WAN Manager cluster on the standby Cisco SD-WAN Manager node to which you copied this backup.

Note

The restore operation does not restore all the information included in the configuration database. Cisco SD-WAN Manager configurations such as users and repositories must be configured on the standby Cisco SD-WAN Manager node after the configuration database is restored using the backup. When you complete the steps that follow, the previously active Cisco SD-WAN Manager nodes cannot be reused. To reuse the nodes, you must perform additional steps that are beyond the scope of this document.

1. On the CLI of the standby Cisco SD-WAN Manager node, run the following command: request nms configuration-db restore path file-path

   In the following example, the configuration database is restored using the backup file db_backup.tar.gz.

   Standby-vManage# request nms configuration-db restore path /home/admin/db_backup.tar.gz
   Configuration database is running in a standalone mode
   Importing database...Successfully restored database

2. Verify that the appropriate services are running on the standby Cisco SD-WAN Manager nodes: On the CLI of each standby Cisco SD-WAN Manager node, run the request nms all status command. From the command output, verify the services running on the node.

3. Verify that every standby Cisco SD-WAN Manager node has a list of all the active and standby Cisco SD-WAN Manager nodes.

   1. From the Cisco SD-WAN Manager menu, choose Configuration > Devices > Controllers.

      Note	Starting from Cisco IOS XE Catalyst SD-WAN Release 17.13.1a, the Controllers tab is renamed as the Control Components tab to stay consistent with Cisco Catalyst SD-WAN rebranding.

   2. Verify that the page displays all active and standby Cisco SD-WAN Manager nodes.

4. On the standby Cisco SD-WAN Manager nodes, enable the transport interface on VPN 0.

   Use one of the following two methods:

   1. Enable the transport interface in VPN 0: On the CLI of each standby Cisco SD-WAN Manager node, run the no shutdown command.

      Active-vManage# config
      Active-vManage(config)# vpn 0 interface interface-name
      Active-vManage(config-interface)# no shutdown
      Active-vManage(config-interface)# commit and-quit

   2. Activate the tunnel interface in VPN 0: On the CLI of each standby Cisco SD-WAN Manager node, run the tunnel-interface command.

      Active-vManage# config
      Active-vManage(config)# vpn 0 interface interface-name
      Active-vManage(config-interface)# tunnel-interface
      Active-vManage(config-interface)# commit and-quit

5. Add each standby Cisco SD-WAN Manager node to the overlay network.

   1. From the Cisco SD-WAN Manager menu, choose Configuration > Devices.

   2. Click Controllers.

      Note	Starting from Cisco IOS XE Catalyst SD-WAN Release 17.13.1a, the Controllers tab is renamed as the Control Components tab to stay consistent with Cisco Catalyst SD-WAN rebranding.

   3. For a Cisco SD-WAN Validator, click ... and click Edit.

   4. In the Edit dialog box, enter the following details of the Cisco SD-WAN Validator: WAN transport IP address, username, and password.

   5. Repeat Step 5c and Step 5d for every Cisco SD-WAN Validator.

6. Disconnect the active Cisco SD-WAN Manager nodes from the overlay network.

   Note	In a lab environment, where you are simulating a disaster scenario, you can perform this step. However, if you cannot reach Cisco SD-WAN Manager instances in an actual disaster scenario, you may not be able to perform this step and can omit the step.

   Use one of the following two methods:

   1. Shut down the transport interface in VPN 0: On the CLI of each active Cisco SD-WAN Manager node, run the shutdown command.

      Active-vManage# config
      Active-vManage(config)# vpn 0 interface interface-name
      Active-vManage(config-interface)# shutdown
      Active-vManage(config-interface)# commit and-quit

   2. Deactivate the tunnel interface in VPN 0: On the CLI of each active Cisco SD-WAN Manager node, run the no tunnel-interface command.

      Active-vManage# config
      Active-vManage(config)# vpn 0 interface interface-name
      Active-vManage(config-interface)# no tunnel-interface
      Active-vManage(config-interface)# commit and-quit

7. From the standby Cisco SD-WAN Manager, send the updated controller and device list to the Cisco SD-WAN Validator.

   Send the list of controllers:

   1. From the Cisco SD-WAN Manager menu, choose Configuration > Certificates.

   2. Click Controllers.

      Note	Starting from Cisco IOS XE Catalyst SD-WAN Release 17.13.1a, the Controllers tab is renamed as the Control Components tab to stay consistent with Cisco Catalyst SD-WAN rebranding.

   3. Click Send to Validator.

      Wait for the configuration task to complete. When the task is complete,

      • The standby Cisco SD-WAN Manager nodes become the active Cisco SD-WAN Manager nodes.

      • The previously active Cisco SD-WAN Manager nodes are no longer part of the overlay network.

      • The active Cisco SD-WAN Manager nodes have the configuration from the most recent configuration database backup.

      • Every controller establishes connection with the other controllers in the network.

   4. Click WAN Edge List.

   5. Click Send to Controllers.

8. Verify that the following are intact:

   • Policies

   • Templates

   • Controller and WAN edge device lists

9. Verify the valid Cisco SD-WAN Manager nodes.

   1. Log in to the CLI of each Cisco SD-WAN Validator and run the show orchestrator valid-vmanage-id command.

      In the command output, verify that the chassis numbers of the active and previously active Cisco SD-WAN Manager nodes are listed.

   2. Log in to the CLI of a WAN edge device and run the show control valid-vmanage-id command.

      In the command output, verify that the chassis numbers of the active and previously active Cisco SD-WAN Manager nodes are listed. Also, check whether the device is connected to the active Cisco SD-WAN Manager nodes and the Cisco Catalyst SD-WAN Controller.

10. Invalidate the previously active Cisco SD-WAN Manager nodes.

    Note	After you invalidate the Cisco SD-WAN Manager nodes, the nodes cannot be reused without performing additional steps that are beyond the scope of this document.

    1. From the Cisco SD-WAN Manager menu, choose Configuration > Certificates.

    2. Click Controllers.

       Note	Starting from Cisco IOS XE Catalyst SD-WAN Release 17.13.1a, the Controllers tab is renamed as the Control Components tab to stay consistent with Cisco Catalyst SD-WAN rebranding.

    3. For each previously active Cisco SD-WAN Manager node, click ... and click Invalidate.

11. Verify the valid Cisco SD-WAN Manager nodes.

    1. Log in to the CLI of each Cisco SD-WAN Validator and run the show orchestrator valid-vmanage-id command.

       In the command output, verify that the chassis numbers of only the active Cisco SD-WAN Manager nodes are listed.

    2. Log in to the CLI of WAN edge device and run the show control valid-vmanage-id command.

       In the command output, verify that the chassis numbers of only the active Cisco SD-WAN Manager nodes are listed. Also, check whether the device is connected to the active Cisco SD-WAN Manager nodes and the Cisco Catalyst SD-WAN Controllers.

The Cisco SD-WAN Manager cluster that was initially the standby cluster is now the active Cisco SD-WAN Manager cluster.