Upgrade Cisco Crosswork

This chapter contains the following topics:

Cisco Crosswork 4.0 to 4.1 Upgrade Workflow

This section provides the high-level workflow for upgrading Cisco Crosswork from release 4.0 to release 4.1. This includes upgrading Cisco Crosswork cluster, Cisco Crosswork Data Gateway and Crosswork Applications to Release 4.1, within a single maintenance window.

Each stage in this upgrade workflow must be executed in sequence, and is explained in detail in later sections of this chapter. The stages are:

  1. Shut Down Cisco Crosswork Data Gateway 2.0 VMs

  2. Create Backup and Shut Down Cisco Crosswork 4.0

  3. Install the Cisco Crosswork 4.1 Cluster


    Note

    While the cluster installation is in progress, you must upgrade NSO to version 5.5.2.12. The process to upgrade NSO is not covered in this document. For more information, see the documentation for Cisco NSO 5.5.2.12. Additionally, if you are planning to use Cisco Optimization Engine individually or as part of the Cisco Network Controller solution, upgrade your SR-PCE to the supported version as mentioned in the Crosswork Network Controller Release Notes.

  4. Install Cisco Crosswork 4.1 Applications

  5. Migrate the Cisco Crosswork 4.0 backup to Cisco Crosswork 4.1

  6. Upgrade to Cisco Crosswork Data Gateway 3.0

  7. Post-upgrade Checklist

The time taken for the entire upgrade window can vary based on size of your deployment profile and the performance characteristics of your hardware.


Warning

Migration of Cisco Crosswork from 4.0 to 4.1 has the following limitations:

  • Third-party device configuration in Device Lifecycle Management (DLM) and Cisco NSO is not migrated, and needs to be re-applied on the new Cisco Crosswork version post migration.

  • Custom user roles (Read-Write/Read) created in Cisco Crosswork 4.0 are not migrated, and need to be updated manually on the new version post migration.

  • Crosswork Health Insights KPI alert history is not retrieved as part of the migration.

Crosswork applications can be independently updated from the Cisco Crosswork UI in case of minor updates or patch releases. For more information, see Update a Crosswork Application (standalone activity).

Shut Down Cisco Crosswork Data Gateway 2.0 VMs

This is the first stage of the upgrade workflow.


Note

When Crosswork Data Gateway VMs are shut down, the data will not be forwarded to data destinations. Check with the application providers to determine if any steps are needed to avoid alarms or other problems.

Before you begin

Take screenshots of the all the tabs in the Data Gateway Management page to keep a record of the list of Crosswork Data Gateways, Attached Device Count in the Cisco Crosswork 4.0 UI. In the Pools tab, for each pool listed here, take a screenshot to make a note of the active, spare, and unassigned VM in the pool. This information is useful during Upgrade to Cisco Crosswork Data Gateway 3.0.

Procedure

Step 1

Check and confirm that all the VMs are healthy and running in your cluster.

Step 2

Shut down the Cisco Crosswork Data Gateway 2.0 VMs.

  1. Log in to the Crosswork Data Gateway VM. See Access Crosswork Data Gateway VM from SSH.

    Crosswork Data Gateway launches an Interactive Console after you login successfully.

  2. Choose 5 Troubleshooting.

  3. From the Troubleshooting menu, choose 5 Shutdown VM to shut down the VM.

Create Backup and Shut Down Cisco Crosswork 4.0

This is the second stage of the upgrade workflow. Creating a backup is a prerequisite when upgrading your Cisco Crosswork to a new software version.


Note

We recommend that you create a backup only during a scheduled upgrade window. Users should not attempt to access Cisco Crosswork while the backup operation is running.

Before you begin

Follow these guidelines whenever you create a backup:

  • Cisco Crosswork will back up the configuration of the system to an external server using SCP. Before you begin you need to have the following configuration in place and information about the SCP server available:

    • The hostname or IP address and the port number of a secure SCP server.

    • A preconfigured path on the SCP server where the backup will be stored.

    • User credentials with file read and write permissions to the directory.

    • The SCP server storage requirements will vary slightly but you must have at least 25 GB of storage.

  • Ensure that you have configured a destination SCP server to store the backup files. This configuration is a one-time activity.

  • Both the Cisco Crosswork cluster and the SCP server must be in the same IP environment. For example: If Cisco Crosswork is communicating over IPv6, so must the backup server.

  • Keep a record of the list of Crosswork applications you have installed in the current version of Cisco Crosswork, as you can only install those applications after migrating to the new version of Cisco Crosswork.

  • If you have onboarded a custom MIB package in Cisco Crosswork 4.0, download a copy of the package to your system. You will need to upload the package after you complete migrating to Cisco Crosswork 4.1. See Post-upgrade Checklist for more infomation.

  • If you have modified Cisco Crosswork 4.0 to include third-party device types, you must download the third-party device configuration file, and re-apply it to Cisco Crosswork 4.1. The device configuration file is located on the cluster node (at /mnt/cw_glusterfs/bricks/brick3/sys-oids.yaml) and on the pod (at /mnt/backup/sys-oids.yaml).

  • If Cisco Crosswork Optimization Engine has feature packs (Local Congestion Mitigation (LCM), Bandwidth Optimization (BWOpt), and Bandwidth on Demand (BWoD)) that are enabled, you must disable them before proceeding. You must also, if available, export the current list of interfaces managed by LCM or BWOpt (Traffic Engineering > Local Congestion Mitigation or Bandwidth Optimization > Link Management > Export icon). Follow the steps documented in "Upgrade Crosswork Optimization Engine Feature Packs" in the latest Cisco Crosswork Optimization Engine Release Notes.

Procedure

Step 1

Check and confirm that all the VMs are healthy and running in your cluster.

Step 2

Configure an SCP backup server:

  1. From the Cisco Crosswork 4.0 main menu, choose Administration > Backup and Restore.

  2. Click Destination to display the Edit Destination dialog box. Make the relevant entries in the fields provided.

  3. Click Save to confirm the backup server details.

Step 3

Create a backup:

  1. From the Cisco Crosswork 4.0 main menu, choose Administration > Backup and Restore.

  2. Click Actions > Backup to display the Backup dialog box with the destination server details prefilled.

  3. Provide a relevant name for the backup in the Job Name field.

  4. If any of the VMs or applications are not in Healthy state, but you want to create the backup, check the Force check box.

    Note 

    The Force option must be used only after consultation with the Cisco Customer Experience team.

  5. Uncheck the Backup NSO checkbox if you don't want to include Cisco NSO data in the backup.

    If you do want to include Cisco NSO data in the Cisco Crosswork backup process, follow the instructions given in Backup Cisco Crosswork with Cisco NSO section in the Cisco Crosswork Infrastructure 4.1 and Applications Administration Guide instead of the instructions here.

  6. Complete the remaining fields as needed.

    If you want to specify a different remote server upload destination: Edit the pre-filled Host Name, Port, Username, Password and Remote Path fields to specify a different destination.

  7. (Optional) Click Verify Backup Readiness to verify that Cisco Crosswork has enough free resources to complete the backup. Cisco Crosswork will also confirm that none of the applications are being updated, if the remote destination is correctly defined and the if applications are healthy. If the verifications are successful, Cisco Crosswork displays a warning about the time-consuming nature of the operation. Click OK.

    If the verification is unsuccessful, please contact the Cisco Customer Experience team for assistance.

  8. Click Start Backup to start the backup operation. Cisco Crosswork creates the corresponding backup job set and adds it to the job list. The Job Details panel reports the status of each backup step as it is completed.

  9. To view the progress of a backup job: Enter the job details (such as Status or Job Type) in the search fields in the Backup and Restore Job Sets table. Then click on the job set you want.

    The Job Details panel displays information about the selected job set, such as the job Status, Job Type, and Start Time. If there’s a failed job, hover the mouse pointer over the icon near the Status column to view the error details.

    Note 

    If you do not see your backup job in the list, refresh the Backup and Restore Job Sets table.

  10. If the backup fails during upload to the remote server: In the Job Details panel, just under the Status icon, click the Upload backup button to retry the upload.

    Note 

    Upload can fail due to connectivity problems with the SCP backup server (for example, incorrect credentials, missing directory or directory permissions, missing path and so on). This is indicated by failure of the task uploadBackupToRemote). If this happens, check the SCP server details, correct any mistakes and try again. Alternatively, you can use the Destination button to specify a different SCP server and path before clicking Upload backup.

Step 4

After a successful backup, shut down the Cisco Crosswork cluster by powering down the VMs hosting each node (start with the Hybrid VMs):

  1. Log into the VMware vSphere Web Client.

  2. In the Navigator pane, right-click the VM that you want to shut down.

  3. Choose Power > Power Off.

  4. Wait for the VM status to change to Off.

  5. Wait for 30 seconds and repeat steps 4a to 4d for each of the remaining VMs.
Step 5

Move Cisco NSO into read-only mode to avoid any unintended updates to Cisco NSO during the upgrade.

Use the following command to move NSO to read-only mode:
ncs_cmd -c maapi_read_only
For more information, please refer to the documentation for Cisco NSO 5.4.2.

Install the Cisco Crosswork 4.1 Cluster

This is the third stage of the upgrade workflow. After the successful backup of Cisco Crosswork 4.0, proceed to install Cisco Crosswork 4.1 cluster.


Note

The number of nodes installed in Cisco Crosswork 4.1 must be equal or more than the number of nodes in Cisco Crosswork 4.0.

Before you begin

Procedure

Step 1

Install Cisco Crosswork 4.1 cluster using any of the installation methods described in Install the Crosswork Cluster.

Note 

During installation, Cisco Crosswork will create a special administrative ID (virtual machine (VM) administrator, with the username cw-admin, and the default password cw-admin). The administrative username is reserved and cannot be changed. The first time you log in using this administrative ID, you will be prompted to change the password. Data center administrators use this ID to log into and troubleshoot the Crosswork application VM. You will use it to verify that the VM has been properly set up.

Step 2

After the installation is completed, log into the Cisco Crosswork UI and check if all the nodes are up and running in the cluster.

  1. From the Cisco Crosswork main menu, choose Administration > Crosswork Manager > Crosswork Summary.

  2. Click Crosswork Cluster tile to view the details of the cluster such as resource utilization by node, the IP addresses in use, whether each node is a hybrid or worker, and so on.

Install Cisco Crosswork 4.1 Applications

This is the fourth stage of the upgrade workflow. After the successful installation of Cisco Crosswork 4.1 cluster, proceed to install Cisco Crosswork 4.1 applications.


Note

You can only install 4.1 versions of the Cisco Crosswork applications that were backed up during Create Backup and Shut Down Cisco Crosswork 4.0.

Procedure

Step 1

Install Cisco Crosswork 4.1 applications using the steps described in Install Crosswork Applications.

Step 2

After the applications are successfully installed, check the health of the Cisco Crosswork 4.1 cluster.

  1. From the Cisco Crosswork main menu, choose Administration > Crosswork Manager > Crosswork Summary.

  2. Click Crosswork Cluster tile to view the health details of the cluster.

Migrate the Cisco Crosswork 4.0 backup to Cisco Crosswork 4.1

This is the fifth stage of the upgrade workflow. After the successfully installing Cisco Crosswork 4.1 applications, proceed to migrate the backup of Cisco Crosswork 4.0 on Cisco Crosswork 4.1 cluster.

Before you begin

Before you begin, ensure that you have:

  • The hostname or IP address and the port number of a secure SCP server.

  • The name and path of the backup file created in Create Backup and Shut Down Cisco Crosswork 4.0.

  • User credentials with file read and write permissions to the directory.

Procedure

Step 1

Configure an SCP backup server:

  1. From the main menu, choose Administration > Backup and Restore.

  2. Click Destination to display the Edit Destination dialog box.

  3. Make the relevant entries in the fields provided.

    Note 

    In the Remote Path field, please provide the location of the backup created in Create Backup and Shut Down Cisco Crosswork 4.0.

  4. Click Save to confirm the backup server details.

Step 2

Migrate the Cisco Crosswork 4.0 backup on the Cisco Crosswork 4.1 cluster:

  1. From the Cisco Crosswork main menu, choose Administration > Backup and Restore.

  2. Click Actions > Data Migration to display the Data Migration dialog box with the destination server details prefilled.

  3. Provide the name of the data migration backup (created in Create Backup and Shut Down Cisco Crosswork 4.0) in the Backup File Name field.

  4. If you want to perform the data migration backup despite any Cisco Crosswork application or microservice issues, check the Force check box.

  5. Click Start Migration to start the data migration operation. Cisco Crosswork creates the corresponding data migration job set and adds it to the Backup and Restore Job Sets table. The Job Details panel reports the status of each backup step as it is completed.

    Note 

    If you do not see your job in the list, refresh the Backup and Restore Job Sets table.

  6. To view the progress of a data migration job: Enter the job details (such as Status or Job Type) in the search fields in the Backup and Restore Job Sets table. Then click on the job set you want.

    The Job Details panel displays information about the selected job set, such as the job Status, Job Type, and Start Time. If there’s a failed job, hover the mouse pointer over the icon near the Status column to view the error details.

    Note 

    Crosswork UI and Grafana monitoring might become temporarily unavailable during the data migration operation.

  7. If the data migration fails in between, you need to restart the procedure from step 1.

Step 3

After the data migration is successfully completed, check the health of the Cisco Crosswork 4.1 cluster.

  1. From the Cisco Crosswork main menu, choose Administration > Crosswork Manager > Crosswork Summary.

  2. Click Crosswork Cluster tile to view the health details of the cluster.

Upgrade to Cisco Crosswork Data Gateway 3.0

This is the final stage of the Crosswork 4.0 to Crosswork 4.1 upgrade workflow. Before you proceed, ensure that you have completed all the steps from the previous stages in the upgrade workflow.


Note

This procedure is required only for a Cisco Crosswork Data Gateway Base VM upgrade. Upgrade of other components, such as collectors, is performed by Cisco Crosswork.

Cisco Crosswork Data Gateway functions as a passive device in the network. The Crosswork Data Gateway upgrade process consists of shutting down the Crosswork Data Gateway 2.0 VMs and replacing this with the Crosswork Data Gateway 3.0 VMs.

Pools and device mapping information are migrated to the 3.0 VMs by running the Migration Utility API:

https://<VIP>:30603/crosswork/inventory/v1/dg/vdg/migrate

The Migration Utilty API in DLM is an accumulative API. You can safely run it multiple times.

Procedure

Step 1

Install new Cisco Crosswork Data Gateway 3.0 VMs with the same number and the same information (management interface importantly) as the Crosswork Data Gateway 2.0 VMs. Follow the steps in the Install Cisco Crosswork Data Gateway.

Step 2

Ensure that the new Cisco Crosswork Data Gateway VMs have enrolled with Cisco Crosswork and have the Administration state Up and Operational state as Not Ready. See Cisco Crosswork Data Gateway Authentication and Enrollment.

Step 3

Move Cisco NSO out of maintenance or read-only mode. For more information, see Related Documentation for 5.5.2.9.

ncs_cmd -c maapi_read_write
Step 4

(Optional) If you have onboarded a custom MIB package in Cisco Crosswork 4.0, upload the custom MIB package that you had downloaded (as instructed in Create Backup and Shut Down Cisco Crosswork 4.0). For information on how to do this, see Section: Add a Custom Software Package in the Cisco Crosswork Infrastructure 4.1 Applications and Administration Guide. After uploading the custom MIB package, do the following checks:

  • Restart robot-alerting, robot-fleet and pulse micro-services.

  • Disable all the KPIs which were using the custom MIB package.

  • After the jobs are successfully disabled, enable all the KPIs that are using custom KPIs.
Step 5

Fetch the JWT token to run the Migration Utility API.

Note 
You can use any tool to perform the API calls. For the purpose of these instructions, we have used POSTMAN.

  1. Run the following API to get the TGT.

    https://<VIP>:30603/crosswork/sso/v1/tickets

    
HTTP method: POST
Headers:
    Content-Type: application/x-www-form-urlencoded
    Accept: text/plain
Body:
    username=<Cisco Crosswork UI login username>
    password-<Cisco Crosswork UI login password>

  2. Get the JWT after getting the TGT.

    https://<VIP>:30603/crosswork/sso/v2/tickets/jwt

    
HTTP method:POST
Headers:
    Content-Type: application/x-www-form-urlencoded
    Accept: text/plain
Body:
    service=https://<VIP>:30603/app-dashboard
    tgt=<TGT from step a>
Step 6

Create Crosswork Data Gateway pools by executing the Migration utility API.

API: https://<VIP>:30603/crosswork/inventory/v1/dg/vdg/migrate
HTTP method:POST
Headers:
    Authorization:Bearer <JWT from step 4b>
    Content-Type: application/json
Body:
{} //empty json needs to be sent

Note 

The API response will always have the status code as 200. The API response body contains a full report with the following details:

  • Crosswork Data Gateway Pools that have been created successfully.

  • Crosswork Data Gateway Pools that have not been created and the reason they have not been created.

  • Crosswork Data Gateway Pools that already exist and are ready for device migration.

Copy the report that is returned inside the Migration Utility API. This report is useful during troubleshooting in case there are issues.

Step 7

Verify that all Crosswork Data Gateway Pools have been created.

  1. Navigate to Administration > Data Gateway Management in Cisco Crosswork UI.

  2. Verify that all the Crosswork Data Gateway Pools from Cisco Crosswork 4.0, are listed under the Data Gateways tab.

  3. In the Pools tab, edit each Crosswork Data Gateway pool to verify that the active Crosswork Data Gateway is same as one that you noted in Cisco Crosswork 4.0.

    For example, the Crosswork Data Gateway pool in the following image contains two VMs, where the active VM is 172.23.247.78

    Note 

    If there issues such as, Crosswork Data Gateway pool has not been created or a different VM is selected as active instead of the VM that was active in Cisco Crosswork 4.0 deployment, check for the issue in the API report generated in response to the API call. Refer to the Section: Troubleshoot Crosswork Data Gateway Upgrade Issues for troubleshooting and suggested workarounds for the issue.

Step 8

Attach devices to Crosswork Data Gateways 3.0 using Migration Utility API

Map the devices from Crosswork Data Gateways 2.0 to the newly created Crosswork Data Gateways 3.0 in Cisco Crosswork 4.1 by running the Migration Utility API (as explained earlier in Step 5).

Running the Migration Utility API this time validates the Crosswork Data Gateways and attaches all devices to the corresponding Crosswork Data Gateways from Cisco Crosswork 4.0.

Step 9

Verify that devices are attached to the Crosswork Data Gateways 3.0 in the Cisco Crosswork 4.1 UI.

  1. Navigate to the Administration > Data Gateway Management page.

  2. Check the Attached Device Count of the Crosswork Data Gateway.

    Note 
    In case of issues (such as a missing Crosswork Data Gateway pool or a pool that does not have any devices attached to it), see Section: Troubleshoot Crosswork Data Gateway Upgrade Issues.

Troubleshoot Crosswork Data Gateway Upgrade Issues

The following table lists common problems that might be experienced when upgrading the Crosswork Data Gateway, and provides approaches to identifying the source of the problem and solving it.

Issue Symptom Recommended Action

1. Forgot to enroll one or more Crosswork Data Gateway 3.0 VMs.

 One of the Crosswork Data Gateway pools has not been created.

Enroll the missing Crosswork Data Gateways and repeat the data migration steps in (Step 5 onwards) in the section Upgrade to Cisco Crosswork Data Gateway 3.0.

2. Some of the Crosswork Data Gateway VMs were in Error or Degraded state when executing the migration procedure

One of the Crosswork Data Gateway pools has not been created.

Wait for the Crosswork Data Gateway VMs to have the state as Up or Not Ready state. Take action if necessary to get the VM to Not Ready state. Repeat the Crosswork Data Gateway data migration steps as described (Step 5 onwards) in the section Upgrade to Cisco Crosswork Data Gateway 3.0.

3. Crosswork Data Gateway pool has been created with the correct VMs, but a different VM is selected as active from the one that was active in the Cisco Crosswork 4.0 deployment.

A different VM is selected as active in the Crosswork Data Gateway pool.

  1. Edit the Crosswork Data Gateway Pool to remove all VMs except the one that should be active per the Cisco Crosswork 4.0 deployment and save the pool.

  2. Edit the Crosswork Data Gateway pool again to add back all the VMs you removed from the pool and save the pool.

4. Crosswork Data Gateway does not have any devices attached to it even after running the VDG migration utility multiple times.

No devices are attached to the Crosswork Data Gateway.

  1. Edit the Crosswork Data Gateway Pool to remove any VMs that were defined as Standy in Cisco Crosswork 4.0 deployment and save the pool.

  2. Repeat Step 6 and Step 7 as described in the section Upgrade to Cisco Crosswork Data Gateway 3.0.

  3. Edit the Crosswork Data Gateway pool and add back the standby VMs to the pool and save the pool.

Post-upgrade Checklist

After the upgrade to Cisco Crosswork 4.1 is completed, check the health of the new cluster. If your cluster is healthy, perform the following activities:

  • Navigate to Administration > Collection Jobs in Cisco Crosswork 4.1 UI and delete the duplicate system jobs.

  • Verify that the collection jobs are running on the Crosswork Data Gateway 3.0 VMs in the Administration > Collection Jobs page. At this point, you can delete the 2.0 VMs.

  • Verify the restored AAA data by logging in using default credentials, and configure custom user roles (Read-Write/Read) in Cisco Crosswork 4.1.

  • (Optional) Based on your network requirements, download the relevant map files from cisco.com and re-upload them to Cisco Crosswork 4.1.

  • (Optional) If any NSO device onboarding policy was set in Cisco Crosswork 4.0, you must update the policy with new Network Element Drivers (NED) on the NSO.

  • (Optional) Re-apply any third-party device configurations (used in Cisco Crosswork 4.0) to Cisco Crosswork 4.1.

  • If you are using Crosswork Optimization Engine, perform the following actions:

If you encounter errors in any of the above activities, please contact the Cisco Customer Experience team.

Update a Crosswork Application (standalone activity)

This section explains how to independently update a Crosswork application from the Cisco Crosswork UI in case of minor updates or patch releases. This procedure is not part of the upgrade workflow discussed in the earlier sections.

Before you begin, ensure that you:

  • Take a backup of your data (using the backup/restore functionality) before any critical upgrade.

  • Download the latest version of the Crosswork APPlication file (CAPP) from cisco.com to your local machine.


Note

Crosswork does not support the downgrade operation of a CAPP file. However, if you want to go back to an older application version, you can uninstall the application and install the older version of the application. In case of a downgrade, you are advised to take a backup of your data prior to the operation.

Procedure

Step 1

Click on Administration > Crosswork Management, and select the Application Management tab.

The Crosswork Platform Infrastructure and any applications that are added are displayed here as tiles.
Step 2

Click on the Add File (.tar.gz) option to add the application CAPP file that you had downloaded.

Step 3

In the Add File dialog box, enter the relevant information and click Add.

Once the CAPP file is added, you can observe the existing application tile (in this example, Zero Touch Provisioning) displaying an upgrade prompt.

Step 4

To upgrade, click the Upgrade prompt and the new version of the application is installed.

The upgrade progress is displayed on the application tile.
Step 5

Alternately, click on the tile, and select the Upgrade option from the drop down list.

In the Upgrade screen, select the new version that you want to upgrade to, and click Upgrade.

Step 6

(Optional) Click on Job History to see the progress of the upgrade operation.

Note 

During an upgrade operation, typically only the components that have changed between the existing CAPP file and the new CAPP file are installed, as the new version may continue to use the most of the resources of the older version. This ensures a quick operation that happens without disruption to the current system and session.

Note 

During an upgrade, the application that is being updated will be unavailable until the update is completed. During this time, any other users using the application will be notified via an alarm about the upgrade.