Analyze AMP Diagnostic Bundle for High CPU

Introduction

This document describes the steps to analyze a diagnostic bundle from Advanced Malware Protection (AMP) for Endpoints Public Cloud on Windows devices to troubleshoot high CPU usage.

Contributed by Luis Velazquez and Edited by Yeraldin Sánchez, Cisco TAC Engineers.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Access to the AMP console

Components Used

The information in this document is based on these software and hardware versions:

• AMP for Endpoints Console 5.4.20200204
• Windows operating system devices

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Troubleshoot

This section provides information you can use to troubleshoot your configuration.

Verify if another antivirus is installed on the machine

If another AV (antivirus) is installed, ensure the main process of the AV is excluded in the policy configuration

Tip: Use the Cisco-Maintained exclusions if the software that is used is included on the list, remember that these exclusions can be added to new versions of an application.

In order to see the lists available in Cisco maintained exclusions section, navigate to Management > Policies > Edit > Exclusions > Cisco-Maintained Exclusions.
Select the ones your endpoint would need according to the software currently installed on the machine, then, save the policy, as shown in the image.

Identify if the high CPU happens when a specific application is in use

Identify if the issue happens while one application or a few of them are executed if you are able to replicate the issue helps in the process of identifying potential exclusions.

Gather diagnostic bundle for analysis

Enable Debug Log Level

In order to gather a useful diagnostic bundle, the debug log level must be enabled.

Debug Level in the endpoint

If you can replicate the issue and have access to the endpoint, below is the best procedure to capture the diagnostic bundle:

1. Open AMP GUI
2. Navigate to Settings
3. Scroll to the bottom of AMP GUI and open Cisco AMP Connector Settings
4. Click on Enable Debug Logging
5. Debug Logging Status must change to Started. This procedure enables debug level until the next policy heartbeat, by default 15 minutes
   
   

Debug level in the policy

If you don't have access to the endpoint or the issue can't be reproduced consistently, the debug log level must be enabled in the policy.

In order to enable debug log level by policy navigate to Management > Policies > Edit > Advanced Settings > Connector Log Level and Management > Policies > Edit > Advanced Settings > Tray Log Level, then select Debug and save the policy, as shown in the image.

Caution: If debug mode is enabled from the policy, all endpoints receive this change.

Note: Sync the policy of the endpoint to ensure the debug level is applied or wait for the heartbeat interval, by default it is 15 minutes.

Reproduce the issue and gather a diagnostic bundle

When the debug level is configured wait till the state of High CPU happens on the system or manually reproduce the conditions previously identified and then gather the diagnostic bundle.

In order to collect the bundle navigate to C:\Program Files\Cisco\AMP\X.X.X (Where X.X.X is the latest AMP version installed on the system) and run the application ipsupporttool.exe this process creates a .7z file on the desktop named CiscoAMP_Support_Tool_%date%.7z

Note: Connector version 6.2.3 and later can request a bundle remotely, navigate to Management > Computers, expand the endpoint record and use the option Diagnose.

Note: The diagnostic bundle can also run from a CMD prompt with the command: "C:\Program Files\Cisco\AMP\X.X.X\ipsupporttool.exe", or "C:\Program Files\Cisco\AMP\X.X.X\ipsupporttool.exe" -o "X:\Folder\I\Can\Get\To", where X.X.X is the latest AMP version installed, the second command can be used in order to select the output folder for the .7z file.

Make the analysis

There are two ways to analyze a diagnostic file:

• Diag_Analyzer.exe
• Amphandlecount.ps1

Diag_Analyzer.exe

Step 1. Download the application here.

Step 2. In the GitHub page, there is a README file with further instructions on usage.

Step 3. Copy the diagnostic file CiscoAMP_Support_Tool_%date%.7z on the same folder that Diag_Analyzer.exe is located.

Step 4. Execute the application Diag_Analyzer.exe.

Step 5. In the new prompt confirm if you want to get the exclusions from the policy with a Y or an N.

Step 6. The script result contains:

• Top 10 Processes
• Top 10 Files
• Top 10 Extensions
• Top 100 Paths
• All files
  

Note: Diag_Analyzer.exe checks the provided AMP diagnostic file for sfc.exe.log files. then, creates a new directory with the diagnostic file name and store the log files outside of the .7z, in the parent directory of the diagnostic, after this, it parses the logs and determines the top 10 processes, files, extensions, and paths, finally, it prints information to the screen and also to a {Diagnostic}-summary.txt file.

Amphandlecount.ps1

Step 1. Download the script amphandlecounts.txt from the bottom of this community post Review Scanned Files from AMP.

Step 2. In order to run the script in Windows, rename it to amphandlecount.ps1.

Step 3. For convenience copy amphandlecount.ps1 file to a folder of his own.

Step 4. Unzip the CiscoAMP_Support_Tool_%date%.7z file and identify the sfc.log's files on the path CiscoAMP_Support_Tool_2019_06_13_18_26_37\Program Files\Cisco\AMP\X.X.X .

Step 5. Copy the sfc.log's files on the amphandlecount.ps1 folder.

Step 6. Run amphandlecount.ps1 with PowerShell, then a window is opened and depending on the execution policy on the endpoint can ask for permission to run.

Tip: In order to change the execution policy open a Windows PowerShell and use the next commands:
Set the policy to allow unrestricted execution access - Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy Unrestricted
Set the policy to restrict execution access - Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy Restricted

Step 7. Allow the PowerShell to finish (It might take some time, depending on how many sfc.log are in the folder) after the PowerShell finish, four files are created on the folder:

• data.csv
• results.txt
• sorted_results.txt
• terms.txt
  
  

Step8. The 4 new files contain the result of the analysis:

• data.csv: contains the full path of the files scanned and the father process which created/modified/moved the file
• results.txt: contains the list of processes that are scanned by AMP
• sorted_results.txt: contain the list of processes that are scanned by AMP with the most scanned process
• terms.txt: contains the name of processes scanned by AMP

Step 9. Filter the process name with high counts from the sorted_results.txt in the data.csv you can identify the parent process with its full path, and then proceed to add an exclusion to the policy in a custom list if it is trusted.

Processes to look:

1. Cntrl + F on "data.csv" and search
2. Path of the file scanned by AMP
3. Path of the parent process that copy/moved/modified the file

Note: Note: Usually the exclusion is the type "Process: File Scan" with "Child Processes include" for the parent process that is getting the scans:

Note: Here you can find more information related to the best practices to create exclusions. 

Tune Exclusions

Once the processes or paths are identified, you can add them to the exclusion list that is linked to the policy applied on the endpoint, navigate to Management > Exclusions > Exclusion name > Edit, as shown in the image.

Submit the bundle for analysis to TAC

ATS TAC can help to troubleshoot these scenarios, if that is the case, please be ready to provide the next information upon case creation:

• When does this issue start?

• Is there any recent change?

• Does the issue happen with a particular application?

  • If yes, which application?

• Is there other Antivirus on the system?

  • If yes, which antivirus?
• Collect a debug bundle while the issue is reproduced:
  • Steps to collect a debug bundle