Upgrade Cisco ISE-PIC

Cisco ISE-PIC Upgrade Overview

Upgrading a Cisco ISE-PIC deployment is a multi-step process and must be performed in the order specified in this document. Upgrade is expected to take approximately 240 minutes + 60 minutes for every 15 GB of data.

Factors that may affect upgrade time include the number of:

  • Endpoints and users in your network

  • Logs in the primary node

You must use the Cisco ISE upgrade bundle to upgrade Cisco ISE-PIC. You can download the upgrade bundle from Cisco.com.

In order to upgrade your deployment with minimum-possible downtime while providing maximum resiliency and ability to roll back, and minimum errors, perform the upgrade in the following order:

  1. Back up all configuration data before beginning upgrade in order to ensure you can easily roll back manually if necessary.

  2. Choose the upgrade process based on your deployment:

    • Standalone deployment

      1. Upgrade the node. Refer to Upgrade a Standalone Node.

      2. Run upgrade verification and network tests after you upgrade the node. Refer to

        Verify the Upgrade Process.


      Note

      For details about the parts of this step, refer to:


    • High Availability (two nodes) Deployment

      1. Upgrade the secondary node first, keeping the PAN at the previous version until the secondary node upgrade is confirmed, in order to use the PAN for rollback if the initial upgrade fails.

      2. Run upgrade verification and network tests after you upgrade the seconary node.

      3. Upgrade the PAN.

        After upgrading both nodes, the Secondary Administration Node is now the Primary Administration Node, installed with the upgraded version, and the original Primary Administration Node is now the Secondary Administration Node, also installed with the upgraded version.

      4. Re-run the upgrade verification and network tests after you upgrade the Primary Administration Node.

      5. When you finish upgrading the original primary node (the second upgrade), in the Edit Node window from the currently secondary node, click Promote to Primary to promote it to become the Primary Administration Node (as was in your old deployment), if required.

Validate Data to Prevent Upgrade Failures

Cisco ISE-PIC offers an Upgrade Readiness Tool (URT) that you can run to detect and fix any data upgrade issues before you start the upgrade process.

Most of the upgrade failures occur because of data upgrade issues. The URT is designed to validate the data before upgrade to identify, and report or fix the issue, wherever possible.

The URT is available as a separate downloadable bundle that can be run on a Secondary Administration Node, for high availability, or on the Standalone Node for a single-node deployment. No downtime is necessary when running this tool.


Warning

In multiple-node deployments, do not run the URT on the Primary Administration Node.


You can run the URT from the Command-Line Interface (CLI) of the Cisco ISE-PIC node. The URT does the following:

  1. Verifies that the URT is run on a standalone Cisco ISE-PIC node or a Secondary Administration Node

  2. Checks if the URT bundle is less than 45 days old—This check is done to ensure that you use the most recent URT bundle

  3. Checks if all the prerequisites are met.

    The following prerequisites are checked by the URT:

    • Version compatibility

    • Disk space


      Note

      Verify the available disk size with Disk Requirement Size. If you are required to increase the disk size, reinstall ISE and restore a config backup.


    • NTP server

    • Memory

    • System and trusted certificate validation

  4. Clones the configuration database

  5. Copies latest upgrade files to the upgrade bundle


    Note

    If there are no patches in URT bundle then the output will return: N/A. This is an expected behaviour while installing a hot patch.
  6. Performs a schema and data upgrade on the cloned database

    • (If the upgrade on the cloned database is successful) Provides an estimate of time it should take for the upgrade to end.

    • (If the upgrade is successful) Removes the cloned database.

    • (If the upgrade on cloned database fails) Collects the required logs, prompts for an encryption password, generates a log bundle, and stores it in the local disk.

Download and Run the Upgrade Readiness Tool

The Upgrade Readiness Tool (URT) validates the configuration data before you actually run the upgrade to identify any issues that might cause an upgrade failure.

Procedure


Step 1

Create a Repository and Copy the URT Bundle

Step 2

Run the Upgrade Readiness Tool


Create a Repository and Copy the URT Bundle

Create a repository and copy the URT bundle. For information on how to create a repository, see “Create Repositories” in the Chapter “Maintain and Monitor” in the Cisco ISE Administrator Guide.

We recommend that you use FTP for better performance and reliability. Do not use repositories that are located across slow WAN links. We recommend that you use a local repository that is closer to the nodes.

Before you begin

Ensure that you have a good bandwidth connection with the repository.

Procedure

Step 1

Download the URT bundle from Cisco.com. You must use the Cisco ISE URT bundle for Cisco ISE-PIC.

Step 2

Optionally, to save time, copy the URT bundle to the local disk on the Cisco ISE-PIC node.

copy repository_url/path/ise-urtbundle-2.4.0.xxx-1.0.0.SPA.x86_64.tar.gz disk:/

For example, if you want to use SFTP to copy the upgrade bundle, you can do the following:

(Add the host key if it does not exist) crypto host_key add host mySftpserver
copy sftp://aaa.bbb.ccc.ddd/ ise-urtbundle-2.4.0.xxx-1.0.0.SPA.x86_64.tar.gz disk:/

aaa.bbb.ccc.ddd is the IP address or hostname of the SFTP server and ise-urtbundle-2.4.0.xxx-1.0.0.SPA.x86_64.tar.gz is the name of the URT bundle.


Run the Upgrade Readiness Tool

The Upgrade Readiness Tool identifies issues with data that might cause an upgrade failure, and reports or fixes the issues, wherever possible. To run the URT:

Before you begin

Having the URT bundle in the local disk saves time.

Procedure

Enter the application install command to install the URT:

application install ise-urtbundle-filename reponame
Example:
ise/admin# application install ise-urtbundle-2.4.0.x.SPA.x86_64.tar.gz reponame
Save the current ADE-OS running configuration? (yes/no) [yes] ? 
Generating configuration...
Saved the ADE-OS running configuration to startup successfully

Getting bundle to local machine...
Unbundling Application Package...
Verifying Application Signature...
Initiating Application Install...

###########################################
# Installing Upgrade Readiness Tool (URT) #
###########################################

Checking ISE version compatibility
- Successful

Checking ISE persona
- Successful

Along with Administration, other services (MNT,PROFILER,SESSION) are enabled on this node. Installing and running URT might consume additional resources.
Do you want to proceed with installing and running URT now (y/n):y

Checking if URT is recent(<30 days old)
- Successful

Installing URT bundle
- Successful

######################################## 
# Running Upgrade Readiness Tool (URT) # 
######################################## 
This tool will perform following tasks: 
1. Pre-requisite checks 
2. Clone config database 
3. Copy upgrade files 
4. Data upgrade on cloned database 
5. Time estimate for upgrade 

Pre-requisite checks 
==================== 
Disk Space sanity check 
- Successful 
NTP sanity 
- Successful 
Appliance/VM compatibility 
- Successful 
Trust Cert Validation 
- Successful 
System Cert Validation 
- Successful 
Invalid MDMServerNames in Authorization Policies check 
-Successful 
6 out of 6 pre-requisite checks passed
——————
Clone config database
=====================
 [########################################] 100%  Successful                                         

Copy upgrade files
==================
- N/A

Data upgrade on cloned database
===============================
Modifying upgrade scripts to run on cloned database
- Successful

Running schema upgrade on cloned database
- Running db sanity to check and fix if any index corruption
- Auto Upgrading Schema for UPS Model
- Upgrading Schema completed for UPS Model
- Successful

Running sanity after schema upgrade on cloned database
- Successful

Running data upgrade on cloned database
- Data upgrade step 1/97, AuthzUpgradeService(2.0.0.308)... Done in 41 seconds.
- Data upgrade step 2/97, NSFUpgradeService(2.1.0.102)... Done in 1 seconds.
- Data upgrade step 3/97, UPSUpgradeHandler(2.1.0.105)... ..Done in 154 seconds.
- Data upgrade step 4/97, UPSUpgradeHandler(2.1.0.107)... Done in 1 seconds.
- Data upgrade step 5/97, NSFUpgradeService(2.1.0.109)... Done in 0 seconds.
- Data upgrade step 6/97, NSFUpgradeService(2.1.0.126)... Done in 1 seconds.
- Data upgrade step 7/97, NetworkAccessUpgrade(2.1.0.127)... Done in 4 seconds.
- Data upgrade step 8/97, ProfilerUpgradeService(2.1.0.134)... Done in 0 seconds.
- Data upgrade step 9/97, ProfilerUpgradeService(2.1.0.139)... Done in 1 seconds.
- Data upgrade step 10/97, ProfilerUpgradeService(2.1.0.166)... ..Done in 121 seconds.
- Data upgrade step 11/97, NSFUpgradeService(2.1.0.168)... Done in 1 seconds.
- Data upgrade step 12/97, AlarmsUpgradeHandler(2.1.0.169)... Done in 3 seconds.
- Data upgrade step 13/97, RegisterPostureTypes(2.1.0.180)... Done in 2 seconds.
- Data upgrade step 14/97, RegisterPostureTypes(2.1.0.189)... Done in 0 seconds.
- Data upgrade step 15/97, UPSUpgradeHandler(2.1.0.194)... Done in 0 seconds.
- Data upgrade step 16/97, TrustsecWorkflowRegistration(2.1.0.203)... Done in 0 seconds.
- Data upgrade step 17/97, NSFUpgradeService(2.1.0.205)... Done in 0 seconds.
- Data upgrade step 18/97, NetworkAccessUpgrade(2.1.0.207)... Done in 0 seconds.
- Data upgrade step 19/97, NSFUpgradeService(2.1.0.212)... Done in 0 seconds.
- Data upgrade step 20/97, NetworkAccessUpgrade(2.1.0.241)... Done in 2 seconds.
- Data upgrade step 21/97, NetworkAccessUpgrade(2.1.0.242)... Done in 1 seconds.
- Data upgrade step 22/97, UPSUpgradeHandler(2.1.0.244)... Done in 0 seconds.
- Data upgrade step 23/97, ProfilerUpgradeService(2.1.0.248)... Done in 0 seconds.
- Data upgrade step 24/97, NetworkAccessUpgrade(2.1.0.254)... Done in 0 seconds.
- Data upgrade step 25/97, UPSUpgradeHandler(2.1.0.255)... Done in 11 seconds.
- Data upgrade step 26/97, MDMPartnerUpgradeService(2.1.0.257)... Done in 0 seconds.
- Data upgrade step 27/97, NetworkAccessUpgrade(2.1.0.258)... Done in 0 seconds.
- Data upgrade step 28/97, ProfilerUpgradeService(2.1.0.258)... Done in 0 seconds.
- Data upgrade step 29/97, MDMPartnerUpgradeService(2.1.0.258)... Done in 2 seconds.
- Data upgrade step 30/97, UPSUpgradeHandler(2.1.0.279)... Done in 2 seconds.
- Data upgrade step 31/97, NSFUpgradeService(2.1.0.282)... Done in 0 seconds.
- Data upgrade step 32/97, NetworkAccessUpgrade(2.1.0.288)... Done in 0 seconds.
- Data upgrade step 33/97, NetworkAccessUpgrade(2.1.0.295)... Done in 0 seconds.
- Data upgrade step 34/97, CertMgmtUpgradeService(2.1.0.296)... Done in 0 seconds.
- Data upgrade step 35/97, NetworkAccessUpgrade(2.1.0.299)... Done in 0 seconds.
- Data upgrade step 36/97, NetworkAccessUpgrade(2.1.0.322)... Done in 1 seconds.
- Data upgrade step 37/97, NetworkAccessUpgrade(2.1.0.330)... Done in 1 seconds.
- Data upgrade step 38/97, NSFUpgradeService(2.1.0.353)... Done in 0 seconds.
- Data upgrade step 39/97, ProfilerUpgradeService(2.1.0.354)... Done in 0 seconds.
- Data upgrade step 40/97, NSFUpgradeService(2.1.0.427)... Done in 1 seconds.
- Data upgrade step 41/97, NSFUpgradeService(2.1.101.145)... Done in 0 seconds.
- Data upgrade step 42/97, ProfilerUpgradeService(2.1.101.145)... Done in 0 seconds.
- Data upgrade step 43/97, UPSUpgradeHandler(2.1.101.188)... Done in 1 seconds.
- Data upgrade step 44/97, NetworkAccessUpgrade(2.2.0.007)... Done in 0 seconds.
- Data upgrade step 45/97, UPSUpgradeHandler(2.2.0.118)... Done in 5 seconds.
- Data upgrade step 46/97, GuestAccessUpgradeService(2.2.0.124)... Done in 19 seconds.
- Data upgrade step 47/97, NSFUpgradeService(2.2.0.135)... Done in 0 seconds.
- Data upgrade step 48/97, NSFUpgradeService(2.2.0.136)... Done in 1 seconds.
- Data upgrade step 49/97, NetworkAccessUpgrade(2.2.0.137)... Done in 0 seconds.
- Data upgrade step 50/97, NetworkAccessUpgrade(2.2.0.143)... Done in 17 seconds.
- Data upgrade step 51/97, NSFUpgradeService(2.2.0.145)... Done in 5 seconds.
- Data upgrade step 52/97, NSFUpgradeService(2.2.0.146)... Done in 2 seconds.
- Data upgrade step 53/97, NetworkAccessUpgrade(2.2.0.155)... Done in 0 seconds.
- Data upgrade step 54/97, CdaRegistration(2.2.0.156)... Done in 1 seconds.
- Data upgrade step 55/97, NetworkAccessUpgrade(2.2.0.161)... Done in 0 seconds.
- Data upgrade step 56/97, UPSUpgradeHandler(2.2.0.166)... Done in 0 seconds.
- Data upgrade step 57/97, NetworkAccessUpgrade(2.2.0.169)... Done in 1 seconds.
- Data upgrade step 58/97, UPSUpgradeHandler(2.2.0.169)... Done in 0 seconds.
- Data upgrade step 59/97, NetworkAccessUpgrade(2.2.0.180)... Done in 0 seconds.
- Data upgrade step 60/97, CertMgmtUpgradeService(2.2.0.200)... Done in 0 seconds.
- Data upgrade step 61/97, NetworkAccessUpgrade(2.2.0.208)... Done in 0 seconds.
- Data upgrade step 62/97, RegisterPostureTypes(2.2.0.218)... Done in 2 seconds.
- Data upgrade step 63/97, NetworkAccessUpgrade(2.2.0.218)... Done in 1 seconds.
- Data upgrade step 64/97, NetworkAccessUpgrade(2.2.0.222)... Done in 0 seconds.
- Data upgrade step 65/97, NetworkAccessUpgrade(2.2.0.223)... Done in 0 seconds.
- Data upgrade step 66/97, NetworkAccessUpgrade(2.2.0.224)... Done in 2 seconds.
- Data upgrade step 67/97, SyslogTemplatesRegistration(2.2.0.224)... Done in 0 seconds.
- Data upgrade step 68/97, ReportUpgradeHandler(2.2.0.242)... Done in 0 seconds.
- Data upgrade step 69/97, IRFUpgradeService(2.2.0.242)... Done in 0 seconds.
- Data upgrade step 70/97, LocalHostNADRegistrationService(2.2.0.261)... Done in 0 seconds.
- Data upgrade step 71/97, NetworkAccessUpgrade(2.2.0.300)... Done in 0 seconds.
- Data upgrade step 72/97, CertMgmtUpgradeService(2.2.0.300)... Done in 1 seconds.
- Data upgrade step 73/97, NSFUpgradeService(2.2.0.323)... Done in 0 seconds.
- Data upgrade step 74/97, NetworkAccessUpgrade(2.2.0.330)... Done in 1 seconds.
- Data upgrade step 75/97, NSFUpgradeService(2.2.0.340)... Done in 0 seconds.
- Data upgrade step 76/97, NetworkAccessUpgrade(2.2.0.340)... Done in 0 seconds.
- Data upgrade step 77/97, NetworkAccessUpgrade(2.2.0.342)... Done in 0 seconds.
- Data upgrade step 78/97, AuthzUpgradeService(2.2.0.344)... Done in 0 seconds.
- Data upgrade step 79/97, RegisterPostureTypes(2.2.0.350)... Done in 38 seconds.
- Data upgrade step 80/97, ProfilerUpgradeService(2.2.0.359)... Done in 0 seconds.
- Data upgrade step 81/97, DictionaryUpgradeRegistration(2.2.0.374)... Done in 19 seconds.
- Data upgrade step 82/97, UPSUpgradeHandler(2.2.0.403)... Done in 0 seconds.
- Data upgrade step 83/97, DictionaryUpgradeRegistration(2.2.0.410)... Done in 0 seconds.
- Data upgrade step 84/97, UPSUpgradeHandler(2.3.0.100)... Done in 20 seconds.
- Data upgrade step 85/97, UPSUpgradeHandler(2.3.0.110)... Done in 1 seconds.
- Data upgrade step 86/97, NetworkAccessUpgrade(2.3.0.145)... Done in 0 seconds.
- Data upgrade step 87/97, NodeGroupUpgradeService(2.3.0.155)... Done in 0 seconds.
- Data upgrade step 88/97, IRFUpgradeService(2.3.0.155)... Done in 0 seconds.
- Data upgrade step 89/97, UPSUpgradeHandler(2.3.0.158)... Done in 0 seconds.
- Data upgrade step 90/97, NetworkAccessUpgrade(2.3.0.178)... Done in 1 seconds.
- Data upgrade step 91/97, NetworkAccessUpgrade(2.3.0.182)... Done in 0 seconds.
- Data upgrade step 92/97, CertMgmtUpgradeService(2.3.0.194)... Done in 4 seconds.
- Data upgrade step 93/97, UPSUpgradeHandler(2.3.0.201)... Done in 0 seconds.
- Data upgrade step 94/97, NSFUpgradeService(2.3.0.233)... Done in 0 seconds.
- Data upgrade step 95/97, ProfilerUpgradeService(2.3.0.233)... Done in 1 seconds.
- Data upgrade step 96/97, GuestAccessUpgradeService(2.3.0.233)... Done in 8 seconds.
- Successful

Running data upgrade for node specific data on cloned database
- Successful

Time estimate for upgrade
=========================
(Estimates are calculated based on size of config and mnt data only. Network latency between PAN and other nodes 
is not considered in calculating estimates)
Estimated time for each node (in mins):
upsdev-vm11(STANDALONE):102



Application successfully installed

In case the application is not installed successfully during the above execution, URT returns the cause of upgrade failure. You need to fix the issues and re-run the URT.


Change VMware Virtual Machine Guest Operating System and Settings

If you are upgrading Cisco ISE nodes on virtual machines, ensure that you change the Guest Operating System to supported Red Hat Enterprise Linux (RHEL) version. To do this, you must power down the VM, update the Guest Operating System, and power on the VM after the change.

RHEL 7 supports only E1000 and VMXNET3 network adapters. Be sure to change the network adapter type before you upgrade.

Firewall Ports that Must be Open for Communication

If you have a firewall that is deployed between your primary Administration node and the secondary node, the following ports must be open before you upgrade:

  • TCP 1521—For communication between the primary administration node .

  • TCP 443—For communication between the primary administration node and secondary nodes.

For a full list of ports that Cisco ISE-PIC uses, see the Cisco ISE Ports Reference.

Back Up Cisco ISE-PIC Configuration and Operational Data from the Primary Administration Node

Obtain a backup of the Cisco ISE-PIC configuration and operational data from the Command Line Interface (CLI). The CLI command is:

backup backup-name repository repository-name {ise-config | ise-operational} encryption-key {hash | plain} encryption-keyname


Note

When Cisco ISE-PIC runs on VMware, VMware snapshots are not supported for backing up ISE-PIC data.

VMware snapshot saves the status of a VM at a given point of time. In a multi-node Cisco ISE-PIC deployment, data in all the nodes are continuously synchronized with the current database information. Restoring a snapshot might cause database replication and synchronization issues. Cisco recommends that you use the backup functionality included in Cisco ISE-PIC for archival and restoration of data.

Using VMware snapshots to back up ISE-PIC data results in stopping Cisco ISE-PIC services. A reboot is required to bring up the ISE-PIC node.


You can also obtain the configuration and operational data backup from the Cisco ISE-PIC Admin Portal. Ensure that you have created repositories for storing the backup file. Do not back up using a local repository. The following repository types are not supported: CD-ROM, HTTP, HTTPS, or TFTP. This is because these repository types are all either read-only or their protocol does not support the file listing.

  1. Choose Administration > Maintenance > Backup and Restore.

  2. Click Backup Now.

  3. Enter the values as required to perform a backup.

  4. Click OK.

  5. Verify that the backup completed successfully.

Cisco ISE-PIC appends the backup filename with a timestamp and stores the file in the specified repository. In addition to the timestamp, Cisco ISE-PIC adds a CFG tag for configuration backups and OPS tag for operational backups. Ensure that the backup file exists in the specified repository.


Note

Cisco ISE-PIC allows you to obtain a backup from an ISE-PIC node (A) and restore it on another ISE-PIC node (B), both having the same hostnames (but different IP addresses). However, after you restore the backup on node B, do not change the hostname of node B because it might cause issues with certificates.


Back Up System Logs from the Primary Administration Node

Obtain a backup of the system logs from the Primary Administration Node from the Command Line Interface (CLI). The CLI command is:

backup-logs backup-name repository repository-name encryption-key { hash | plain} encryption-key name

Check Certificate Validity

The upgrade process fails if any certificate in the Cisco ISE-PIC Trusted Certificates or System Certificates store has expired. Ensure that you check the validity in the Expiration Date field of the Trusted Certificates and System Certificates windows (Administration > System > Certificates > Certificate Management), and renew them, if necessary, before upgrade.

Also check the validity in the Expiration Date field of the certificates in the CA Certificates window (Administration > System > Certificates > Certificate Authority > Certificate Authority Certificates), and renew them, if necessary, before upgrade.

Export Certificates and Private Keys

We recommend that you export:

  • All local certificates (from all the nodes in your deployment) along with their private keys to a secure location. Record the certificate configuration (what service the certificate was used for).

  • All certificates from the Trusted Certificates Store of the Primary Administration Node. Record the certificate configuration (what service the certificate was used for).

Disable Scheduled Backups before Upgrading

You cannot perform deployment changes when running a backup in Cisco ISE-PIC. Therefore, you must disable automatic configurations in order to ensure that they do not interfere with the upgrade. Ensure that you disable the following configurations before you upgrade Cisco ISE:

  • Scheduled Backups—When planning your deployment upgrade, reschedule the backups after the upgrade. You can choose to disable the backup schedules and recreate them after the upgrade.

    Backups with a schedule frequency of once get triggered every time the Cisco ISE-PIC application is restarted. Hence, if you have a backup schedule that was configured to run only a single time, be sure to disable it before upgrade.

Configure NTP Server and Verify Availability

During upgrade, the Cisco ISE-PIC nodes reboot, migrate, and replicate data from the primary administration node to the secondary administration node. For these operations, it is important that the NTP server in your network is configured correctly and is reachable. If the NTP server is not set up correctly or is unreachable, the upgrade process fails.

Ensure that the NTP servers in your network are reachable, responsive, and synchronized during upgrade.

Upgrade a Two-Node Deployment

Use the application upgrade prepare <upgrade bundle name> <repository name> and proceed commands to upgrade a two-node deployment. The upgrade software automatically deregisters the node and moves it to the new deployment. When you upgrade a two-node deployment, you should initially upgrade only the Secondary Administration Node. When the secondary node upgrade is complete, you upgrade the primary node thereafter.

Before you begin

  • Perform an on-demand backup (manually) of the configuration and operational data from the Primary Administration Node.

Procedure


Step 1

Upgrade the secondary node from the CLI.

The upgrade process automatically removes the original secondary node from the deployment and upgrades it. The original secondary node becomes the upgraded primary node when it restarts.

Step 2

Upgrade the original primary node.

The upgrade process automatically registers the original primary node to the deployment and makes it the secondary node in the upgraded environment.

Step 3

Promote the secondary node, to be the primary node in the new deployment.

After the upgrade is completeensure that you run the application configure ise command and choose 5 (Refresh Database Statistics) on the nodes.


What to do next

Verify the Upgrade Process

Upgrade a Standalone Node

You can use the application upgrade <upgrade bundle name> <repository name> command directly, or the application upgrade prepare <upgrade bundle name> <repository name> and application upgrade proceed commands in the specified sequence to upgrade a standalone node.

If you choose to run this command directly, we recommend that you copy the upgrade bundle from the remote repository to the Cisco ISE-PIC node's local disk before you run the command to save time during upgrade.

Alternatively, you can use the application upgrade prepare <upgrade bundle name> <repository name> and application upgrade proceed commands. The application upgrade prepare <upgrade bundle name> <repository name> command downloads the upgrade bundle and extracts it locally. This command copies the upgrade bundle from the remote repository to the Cisco ISE-PIC node's local disk. After you have prepared a node for upgrade, run the application upgrade proceed command to complete the upgrade successfully.

We recommend that you run the application upgrade prepare <upgrade bundle name> <repository name> and application upgrade proceed commands as described below.

Before you begin

Ensure that you have read the instructions in the section.

Procedure


Step 1

Create a repository on the local disk. For example, you can create a repository called "upgrade."

Example:

ise/admin# conf t 
Enter configuration commands, one per line.  End with CNTL/Z.
ise/admin(config)# repository upgrade 
ise/admin(config-Repository)# url disk: 
% Warning: Repositories configured from CLI cannot be used from the ISE web UI and are not replicated to other ISE nodes.
If this repository is not created in the ISE web UI, it will be deleted when ISE services restart.
ise/admin(config-Repository)# exit 
ise/admin(config)# exit 
Step 2

From the Cisco ISE-PIC command line interface (CLI), enter application upgrade prepare <upgrade bundle name> <repository name> command.

This command copies the upgrade bundle to the local repository "upgrade" that you created in the previous step and lists the MD5 and SHA256 checksum.

Example:

ise/admin# application upgrade prepare <upgrade bundle name> <repository name>application upgrade prepare ise-upgradebundle-2.0.x-2.1.x-2.2.x-2.3.x-to-2.4.0.x.SPA.x86_64.tar.gz upgrade  

Getting bundle to local machine...
Unbundling Application Package...
Verifying Application Signature...

Application upgrade preparation successful
Step 3

From the Cisco ISE-PIC CLI, enter the application upgrade proceed command.

Note 

After beginning the upgrade, you can view the progress of the upgrade by logging in via SSH and using the show application status ise command. The following message appears: % NOTICE: Identity Services Engine upgrade is in progress...

Example:

ise/admin# application upgrade proceed
Initiating Application Upgrade...
% Warning: Do not use Ctrl-C or close this terminal window until upgrade completes.
-Checking VM for minimum hardware requirements
STEP 1: Stopping ISE application...
STEP 2: Verifying files in bundle...
-Internal hash verification passed for bundle
STEP 3: Validating data before upgrade...
STEP 4: Taking backup of the configuration data...
STEP 5: Running ISE configuration database schema upgrade...
- Running db sanity to check and fix if any index corruption
- Auto Upgrading Schema for UPS Model
- Upgrading Schema completed for UPS Model
ISE database schema upgrade completed.
% Warning: Sanity test found some indexes missing in CEPM schema. Please recreate missing indexes after upgrade using app configure ise cli
STEP 6: Running ISE configuration data upgrade...
- Data upgrade step 1/30, UPSUpgradeHandler(2.4.0.101)... Done in 50 seconds.
- Data upgrade step 2/30, UPSUpgradeHandler(2.4.0.116)... Done in 0 seconds.
- Data upgrade step 3/30, MachineAuthenticationSettingsRegistration(2.4.0.120)... Done in 0 seconds.
- Data upgrade step 4/30, GuestAccessUpgradeService(2.4.0.126)... Done in 15 seconds.
- Data upgrade step 5/30, RegisterPostureTypes(2.4.0.127)... Done in 1 seconds.
- Data upgrade step 6/30, UPSUpgradeHandler(2.4.0.127)... Done in 0 seconds.
- Data upgrade step 7/30, UPSUpgradeHandler(2.4.0.134)... Done in 0 seconds.
- Data upgrade step 8/30, NSFUpgradeService(2.4.0.140)... Done in 0 seconds.
- Data upgrade step 9/30, NSFUpgradeService(2.4.0.155)... Done in 1 seconds.
- Data upgrade step 10/30, UPSUpgradeHandler(2.4.0.158)... Done in 1 seconds.
- Data upgrade step 11/30, NSFUpgradeService(2.4.0.160)... Done in 0 seconds.
- Data upgrade step 12/30, NSFUpgradeService(2.4.0.161)... Done in 0 seconds.
- Data upgrade step 13/30, NSFUpgradeService(2.4.0.179)... Done in 0 seconds.
- Data upgrade step 14/30, NetworkAccessUpgrade(2.4.0.182)... Done in 1 seconds.
- Data upgrade step 15/30, StorageUpgradeService(2.4.0.183)... Done in 0 seconds.
- Data upgrade step 16/30, DnsHostnameResolutionRegistration(2.4.0.190)... Done in 0 seconds.
- Data upgrade step 17/30, ProfilerUpgradeService(2.4.0.194)... ..Done in 131 seconds.
- Data upgrade step 18/30, CertMgmtUpgradeService(2.4.0.200)... ..Done in 167 seconds.
- Data upgrade step 19/30, NSFUpgradeService(2.4.0.214)... Done in 0 seconds.
- Data upgrade step 20/30, ERSDictionaryRegistration(2.4.0.215)... Done in 0 seconds.
- Data upgrade step 21/30, NetworkAccessUpgrade(2.4.0.216)... Done in 0 seconds.
- Data upgrade step 22/30, ProfilerUpgradeService(2.4.0.227)... Done in 0 seconds.
- Data upgrade step 23/30, ProfilerUpgradeService(2.4.0.228)... Done in 6 seconds.
- Data upgrade step 24/30, ProfilerUpgradeService(2.4.0.229)... Done in 0 seconds.
- Data upgrade step 25/30, NetworkAccessUpgrade(2.4.0.240)... Done in 0 seconds.
- Data upgrade step 26/30, CertMgmtUpgradeService(2.4.0.293)... Done in 7 seconds.
- Data upgrade step 27/30, ProvisioningUpgradeService(2.4.0.299)... Done in 0 seconds.
- Data upgrade step 28/30, NSFUpgradeService(2.4.0.336)... Done in 2 seconds.
- Data upgrade step 29/30, ProfilerUpgradeService(2.4.0.336)... Done in 0 seconds.
- Data upgrade step 30/30, GuestAccessUpgradeService(2.4.0.336)... Done in 26 seconds.
STEP 7: Running ISE configuration data upgrade for node specific data...
STEP 8: Running ISE M&T database upgrade...
M&T Log Processor is not running
ISE database M&T schema upgrade completed.
cat: /opt/oracle/base/admin/cpm10/dpdump/upgradedb*.properties: No such file or directory

Gathering Config schema(CEPM) stats .....
Gathering Operational schema(MNT) stats ....
% NOTICE: The appliance will reboot twice to upgrade software and ADE-OS. During this time progress of the upgrade is visible on console. It could take up to 30 minutes for this to complete.
Rebooting to do Identity Service Engine upgrade...

The upgrade is now complete.


What to do next

Verify the Upgrade Process

Verify the Upgrade Process

We recommend that you run some network tests to ensure that the deployment functions as expected and that users are able to access resources on your network.

If an upgrade fails because of configuration database issues, the changes are rolled back automatically.

Procedure


Perform any of the following options in order to verify whether the upgrade was successful.

  • Check the ade.log file for the upgrade process. To display the ade.log file, enter the following command from the Cisco ISE-PIC CLI: show logging system ade/ADE.log.?

You can grep for STEP to view the progress of the upgrade:

  • info:[application:install:upgrade:preinstall.sh] STEP 0: Running pre-checks
  • info:[application:operation:preinstall.sh] STEP 1: Stopping ISE application...
  • info:[application:operation:preinstall.sh] STEP 2: Verifying files in bundle...
  • info:[application:operation:isedbupgrade-newmodel.sh] STEP 3: Validating data before upgrade...
  • info:[application:operation:isedbupgrade-newmodel.sh] STEP 4: De-registering node from current deployment.
  • info:[application:operation:isedbupgrade-newmodel.sh] STEP 5: Taking backup of the configuration data...
  • info:[application:operation:isedbupgrade-newmodel.sh] STEP 6: Registering this node to primary  of new deployment...
  • info:[application:operation:isedbupgrade-newmodel.sh] STEP 7: Downloading configuration data from primary  of new deployment...
  • info:[application:operation:isedbupgrade-newmodel.sh] STEP 8: Importing configuration data...
  • info:[application:operation:isedbupgrade-newmodel.sh] STEP 9: Running ISE configuration data upgrade for node specific data...
  • info:[application:operation:isedbupgrade-newmodel.sh] STEP 10: Running ISE M&T database upgrade...
  • info:[application:install:upgrade:post-osupgrade.sh] POST ADEOS UPGRADE STEP 1: Upgrading Identity Services Engine software... 
  • info:[application:operation:post-osupgrade.sh] POST ADEOS UPGRADE STEP 2: Importing upgraded data to 64 bit database...
  • Search for this string to ensure that the upgrade is successful:
    Upgrade of Identity Services Engine completed
          successfully.
  • Enter the show version command to verify the build version.
  • Enter the show application status ise command to verify that all the services are running.

Recover from Upgrade Failures

This section describes what you need to do in order to recover if the upgrade fails.

In rare cases, you might have to reimage, perform a fresh install, and restore data. So it is important that you have a backup of Cisco ISE-PIC configuration data before you start the upgrade. It is important that you back up the configuration data although we automatically try to roll back the changes in case of configuration database failures.

Upgrade Failures

Configuration and Data Upgrade Errors

This section describes some of the known upgrade errors and what you must do to recover from them.


Note

You can check the upgrade logs from the CLI or the status of the upgrade from the console. Log in to the CLI or view the console of the Cisco ISE-PIC node to view the upgrade progress. You can use the show logging application command from the Cisco ISE-PIC CLI to view the following logs (example filenames are given in parenthesis):

  • DB Data Upgrade Log (dbupgrade-data-global-20160308-154724.log)

  • DB Schema Log (dbupgrade-schema-20160308-151626.log)

  • Post OS Upgrade Log (upgrade-postosupgrade-20160308-170605.log)


During upgrade, the configuration database schema and data upgrade failures are rolled back automatically. Your system returns to the last known good state. If this is encountered, the following message appears on the console and in the logs:

% Warning: The node has been reverted back to its pre-upgrade state.
error: %post(CSCOcpm-os-1.4.0-205.i386) scriptlet failed, exit status 1
% Application upgrade failed. Please check logs for more details or contact Cisco Technical Assistance Center for support.

If you need to remediate an upgrade failure to get the node back to the original state, the following message appears on the console. Check the logs for more information.

% Warning: Do the following steps to revert node to its pre-upgrade state."
error: %post(CSCOcpm-os-1.4.0-205.i386) scriptlet failed, exit status 1
% Application upgrade failed. Please check logs for more details or contact Cisco Technical Assistance Center for support.

Validation errors are not an actual upgrade failure. Validations errors may occur. For example, you might see this error if the system does not meet the specified requirements. The system returns to the last known good state. If you encounter this error, ensure that you perform the upgrade as described in this document.

STEP 1: Stopping ISE application...
% Warning: Cannot upgrade this node until the standby PAP node is upgraded and running. If standbyPAP is already upgraded 
and reachable ensure that this node is in SYNC from current Primary UI.
Starting application after rollback...
 
% Warning: The node has been reverted back to its pre-upgrade state.
error: %post(CSCOcpm-os-1.4.0-205.i386) scriptlet failed, exit status 1
% Application upgrade failed. Please check logs for more details or contact Cisco Technical Assistance Center for support.

If the ADE-OS or application binary upgrade fails, the following message appears when you run the show application status ise command from the CLI following a reboot. You should reimage and restore the configuration and operational backups.

% WARNING: An Identity Services Engine upgrade had failed. Please consult logs. You have to reimage and restore to previous version.

For any other types of failures (including cancellation of the upgrade, disconnection of the console session, power failure, and so on), you must reimage and restore the backup.

The term, reimage, refers to a fresh installation of Cisco ISE-PIC. Before you reimage, ensure that you generate a support bundle by running the backup-logs CLI command and place the support bundle in a remote repository in order to help ascertain the cause of failure. You must reimage to the old or new version, as follows:

  • Secondary Administration Node—Reimage to the old version and restore the configuration and operational backup.

  • Primary Administration Node—If there are upgrade failures on the PAN, the system usually returns to the last known good state. If the system does not roll back to the old version, you can reimage to the new version, and register with the new deployment.

In case of upgrade failures, before you try to upgrade again:

  • Analyze the logs. Check the support bundle for errors.

  • Identify and resolve the problem by submitting the support bundle that you generated to the Cisco Technical Assistance Center (TAC).


    Note

    You can view the progress of the upgrade by logging in via SSH and using the show application status ise command. The following message appears: % NOTICE: Identity Services Engine upgrade is in progress...


Upgrade Failures during Binary Install

Problem An application binary upgrade occurs after the database upgrade. If a binary upgrade failure happens, the following message appears on the console and ADE.log:
% Application install/upgrade failed with system removing the corrupted install

Solution Before you attempt any roll back or recovery, generate a support bundle by using the backup-logs command and place the support bundle in a remote repository.

To roll back, reimage the Cisco ISE-PIC appliance by using the previous ISO image and restore the data from the backup file. You need a new upgrade bundle each time you retry an upgrade.

  • Analyze the logs. Check the support bundle for errors.

  • Identify and resolve the problem by submitting the support bundle that you generated to the Cisco Technical Assistance Center (TAC).

Roll Back to the Previous Version

In rare cases, you might have to reimage the Cisco ISE-PIC appliance by using the previous version of ISO image and restoring the data from the backup file. After restoring the data, you can register with the old deployment. Hence, we recommend that you back up the Cisco ISE-PIC configuration data before you start the upgrade process.

Sometimes, upgrade failures that occur because of issues in the configuration database are not rolled back automatically. When this occurs, you get a notification stating that the database is not rolled back, along with an upgrade failure message. In such scenarios, you should manually reimage your system, install Cisco ISE, and restore the configuration data.

Before you attempt to rollback or recovery, generate a support bundle by using the backup-logs command, and place the support bundle in a remote repository.

Post-Upgrade Tasks

See the Identity Services Engine Passive Identity Connector (ISE-PIC) Administrator Guide for additional details about each of these tasks.

VMware Virtual Machine Guest Operating System Configuration

Ensure that the Guest Operating System on the VMware virtual machine is set to Red Hat Enterprise Linux (RHEL) 7 and the network adapter is set to E1000 or VMXNET3.


Note

If you are upgrading to Release 2.4 on an ESXi 5.x server (5.1 U2 minimum), you must upgrade the VMware hardware version to 9 before you can select RHEL 7 as the Guest OS.


Clear Browser Cache

After upgrade, ensure that you clear the browser cache, close the browser, and open a new browser session before you access the Cisco ISE-PIC Admin portal.

Supported browsers are:
  • Mozilla Firefox 79 and earlier versions

  • Mozilla Firefox ESR 60.9 and earlier versions

  • Google Chrome 84 and earlier versions

Reconfigure Active Directory Join Points

The Active Directory join point may be lost during upgrade. Log in to the Admin portal and navigate to check if you need to re-configure a join point.

Configure Active Directory Identity Search Attributes

Cisco ISE-PIC identifies users using the attributes SAM, CN, or both with the sAMAccountName attribute as the default attribute.

You can configure Cisco ISE-PIC to use SAM, CN, or both, if your environment requires it. When SAM and CN are used, and the value of the SAMAccountName attribute is not unique, Cisco ISE-PIC also compares the CN attribute value.

To configure attributes for Active Directory identity search:

  1. Choose Providers > Active Directory. In the Active Directory window, click Advanced Tools, and choose Advanced Tuning. Enter the following details:

    • ISE Node—Choose the ISE node that is connecting to Active Directory.

    • Name—Enter the registry key that you are changing. To change the Active Directory search attributes, enter: REGISTRY.Services\lsass\Parameters\Providers\ActiveDirectory\IdentityLookupField

    • Value—Enter the attributes that ISE uses to identify a user:

      • SAM—To use only SAM in the query (this option is the default).

      • CN—To use only CN in the query.

      • SAMCN—To use CN and SAM in the query.

    • Comment—Describe what you are changing, for example: Changing the default behavior to SAM and CN

  2. Click Update Value to update the registry.

    A pop-up window appears. Read the message and accept the change. The AD connector service in ISE restarts.

Configure Reverse DNS Lookup

Ensure that you have Reverse DNS lookup configured for all Cisco ISE-PIC nodes in your two-node deployment from the DNS server(s). Otherwise, you may run into deployment-related issues after upgrade.

Restore Cisco CA Certificates and Keys

Obtain a backup of the Cisco ISE-PIC CA certificates and keys from the Primary Administration Node and restore it on the Secondary Administration Node. This ensures that the Secondary Administration Node can function as the root CA or subordinate CA of an external PKI in case of a PAN failure and you promote the Secondary Administration Node to be the Primary Administration Node.

Reconfigure Mandatory ISE-PIC System Settings

  • Reconfigure e-mail settings, favorite reports, and data purge settings.

  • Check the threshold and/or filters for specific alarms that you need. All the alarms are enabled by default after an upgrade.

Communications, Services, and Additional Information

  • To receive timely, relevant information from Cisco, sign up at Cisco Profile Manager.

  • To get the business impact you’re looking for with the technologies that matter, visit Cisco Services.

  • To submit a service request, visit Cisco Support.

  • To discover and browse secure, validated enterprise-class apps, products, solutions, and services, visit Cisco DevNet.

  • To obtain general networking, training, and certification titles, visit Cisco Press.

  • To find warranty information for a specific product or product family, access Cisco Warranty Finder.

Cisco Bug Search Tool

Cisco Bug Search Tool (BST) is a gateway to the Cisco bug-tracking system, which maintains a comprehensive list of defects and vulnerabilities in Cisco products and software. The BST provides you with detailed defect information about your products and software.

Documentation Feedback

To provide feedback about Cisco technical documentation, use the feedback form available in the right pane of every online document.