The admiral checks for the uptime of services on service status. It raises an alert when this uptime becomes lower than the preconfigured threshold for alerting.

As an example, Rpminstall is a service which is used to install RPMs during deploys, upgrades, patches, and so on. It is configured to generate an admiral alert if its uptime is less than 80% over one hour. If Rpminstall service goes down for a duration longer than the threshold specified above, an admiral alert for Rpminstall is generated with status ACTIVE.

When the service recovers, its uptime percentage starts increasing. When the uptime goes higher than its threshold, the alert auto closes, and its status moves to CLOSED. In the Rpminstall example described above, Rpminstall Admiral Alert will automatically close when its uptime goes over 80% in one hour.

Note

The close of the alert will ALWAYS lag the service becoming normal. This is because the admiral looks at service health over a duration of time. In the above example, since the Rpminstall alert threshold is set to 80% of an hour of uptime, it needs to be up for at least 48 minutes (80% of one hour) before the alert closes.

No action is required to close the alert. This ensures that all ACTIVE admiral alerts indicate a current underlying issue that needs attention.

Note	No dedicated notification is generated when alerts close.

After an alert moves to CLOSED, it will no longer show under ACTIVE alerts. Closed alerts can still be seen on the UI using the filter Status=CLOSED as shown below:

There are two kinds of admiral alerts:

• Individual Admiral Alert

• Summary Admiral Alert

The alerts that are described in the previous section, alerts that are raised for individual services, fall under the Individual Admiral Alert category. The alert text always contains <Service Name> Admiral Alert. This makes it easy to filter individual alerts by service or by the Admiral Alert suffix.

The admiral generates daily Summary Alerts at midnight UTC. They contain a list of currently active alerts and all alerts closed within the last one day. This allows the user to see the overall cluster health reported by admiral in one place. This is also useful to see closed alerts which do not generate a dedicated notification otherwise. If the cluster is healthy and no alerts were closed within the last one day, no summary notifications are generated for that day. This is done to reduce unnecessary notifications and noise.

The Alerts Text in this case is always Admiral Summary. This makes it easy to filter summary alerts as shown in the following figure.

Since admiral alerts generate an individual notification only once per alert, including/excluding or snoozing specific alerts are not needed. Alerts auto close when the service becomes normal for threshold uptime as described above. There is a force close option available to forcibly close an alert. This should normally be used only to remove summary alerts from the UI as individual alerts close automatically.

Warning

Individual alerts should not be closed forcefully. Doing so while the underlying service is still down or its uptime is below its expected threshold will lead to another alert getting raised for the same service on the next admiral processing iteration.

Admiral Alerts are of Type PLATFORM. As such, these alerts can be configured to be sent to various publishers by appropriate connections for Platform Alerts via the configuration page ./configuration. For convenience, the connection is turned on between Platform Alerts and Internal Kafka by default which allows admiral alerts to be seen on the Current Alerts page (go to Investigate > Alerts) without any manual configuration.

Admiral Alerts are also sent to the email address configured under Platform > Cluster Configuration > Admiral Alert Email.

Thus, users can receive admiral notifications even if they don’t have the TAN edge appliance setup. This is similar to Bosun behavior in previous releases.

These email notifications are generated based on the same triggers as the Current Alerts page. Thus, they are sent on alert creation and a daily summary email at midnight UTC. The daily summary email lists all active alerts and those closed within the last 24 hours.

If there are no active alerts and no alerts closed within the last 24 hours, the summary emails are skipped to reduce email noise.

The Secure Workload on-premises cluster bundles a Unified Computing System (UCS) Cisco Integrated Management Controller (CIMC) Host Upgrade Utility (HUU) ISO. The firmware upgrade option on the Cluster Status page can be used to update a physical bare metal to the version of UCS firmware included in the HUU ISO that has been bundled in the Secure Workload RPMs.

A bare metal host can have the firmware update started on it when the status is Active or Inactive as long as the bare metal state is not Initialized or SKU Mismatch. Only one bare metal can have its UCS firmware that is updated at a time. To start the firmware update, the Secure Workload orchestrator state must be Idle. When the UCS firmware update is initiated, some of the UI functionality specific to the Cluster Status page may be temporarily impacted if the consul leader, active orchestrator, or active firmware manager (fwmgr) must be switched to other hosts - these switchovers should occur automatically. During the firmware update, the firmware details for the bare metal being updated will not be displayed and after the update it may take up to 15 minutes for the firmware details to display again in the Cluster Status page. Before starting the firmware update, check the Service Status page to verify that all services are healthy.

When you initiate a firmware update on a bare metal, fwmgr will verify that the update can continue, gracefully power down the bare metal if needed, then login to the CIMC on the bare metal and start the HUU-based firmware update. That HUU-based firmware update process involves booting the bare metal into the HUU ISO, doing the update, rebooting CIMC to activate the new firmware then booting the bare metal back into the HUU ISO to verify the update was completed. The overall update process can take 2+ hours for a G1 bare metal or 1+ hours for a G2 bare metal. When the firmware update process is initiated, the Service Status page may indicate that some services are unhealthy since a bare metal and all the virtual machines running on that bare metal are no longer active in the cluster. When the firmware update completes, it can take an extra 30 minutes for the bare metal to become active in the cluster again and more time may be needed for all services to become healthy again. If services do not recover within two hours after a firmware update, contact a customer service representative.

You can click a bare metal node in the Cluster Status page to expand details about the bare metal. When a firmware update is initiated, you can click the View Firmware Upgrade Logs button to view the status of the firmware update. The log contains the overall status of the firmware update and the status can be one of the following:

• Firmware update has been triggered: The firmware update was requested but has not started yet. During this status fwmgr will be checking to make sure the services required for the firmware update are functional and that CIMC can reach those services.

• Firmware update is running: The firmware update has been started. When a firmware update reaches this state, CIMC and HUU are in control of the update, and the Secure Workload cluster will report the status that it gets from CIMC about the update.

• Firmware update has timed out: This indicates that some process from the firmware update has exceeded the time that we expect it to complete. The overall firmware update process has a 240-minute time limit when it enters the Firmware update is running phase. During the firmware update CIMC may become unreachable when it reboots into the new version, this unreachable state has a timeout of 40 minutes before the firmware update is declared as timed out. When the firmware update has started, the monitoring of that update will time out after 120 minutes.

• Firmware update has failed with an error: This indicates that an error occurred and the firmware update has failed. CIMC usually does not give an indication of success or failure so this state usually indicates an error occurred before the firmware update actually running.

• Firmware update has finished: The firmware update finished without running into any errors or time outs. CIMC usually does not give an indication of success or failure, it is best to verify that the UCS firmware versions are updated when those details become available in the Cluster Status page - it can take up to 15 minutes for those details to become available.

Below the overall status in the View Firmware Upgrade Logs pop-up is an Update progress section that will contain timestamped log messages indicating the progress of the firmware update. When the Rebooting Host In Progress status is displayed in these log messages, CIMC is in control of the update and the cluster is monitoring that update - most log messages after this come directly from CIMC and are only added to the list of log messages if the status of the update changes.

Below the Update progress section of the View Firmware Upgrade Logs pop-up a Component update status section will be shown when CIMC starts providing individual component update statuses. This section summarizes the status of the update of the various UCS components on the bare metal.

A schedule for data backup can be configured using the Data Backup section on the UI. The backups are triggered either once a day and at the scheduled time based on the configured settings or can be configured to run continuously. A successful backup is called a checkpoint. A checkpoint is a point in time snapshot of the cluster’s primary datastores.

A successful checkpoint can be used to restore the data onto another cluster or the same cluster.

The cluster configuration data are always backed up for every checkpoint. Flow and other data contribute to the bulk of the data backed up. Therefore, if configured appropriately, only incremental changes are backed up. Incremental backups help reduce the amount of data pushed to the external storage, which avoids overloading the network. Optionally, a full backup can be triggered on a schedule for all data sources when incremental backup is configured. A full backup copies every object in a checkpoint, even if it is already copied and the object has not changed. This can add significant load on the cluster, on the network between the cluster and the object store, and the object store itself. A full backup may be necessary if there are corruptions in the objects or the object store has any unrecoverable hardware failures. Additionally, if the bucket provided for backup changes, a full backup is automatically enforced since a full backup is necessary before incremental backups will be useful.

Note	The secure connector information is not backed up or restored in the on-premise version of Secure Workload, but is backed up and restored in the SaaS version of Secure Workload. The virtual patch information of FMC connectors is not restored after restoring the backed-up data.

The object store must provide a S3V4 complaint interface.

Note

A few S3V4-compliant object stores do not support the DeleteObjects functionality. The DeleteObjects functionality is required to delete outdated checkpoint information. The lack of this functionality can lead to failures when attempting to delete outdated checkpoints from the storage and can cause the storage to run out of space.

• Location

  The location of the object store is critical to the latency involved in backing up and restoring from the store. To improve restore time, ensure that the object store is located closer to the standby cluster.

• Bucket

  Create a new and dedicated bucket for Secure Workload in the object store. Only the cluster should have write access to this bucket. The cluster will write objects and manage retention on the bucket. Provision at least 200 TB of storage for the bucket and obtain an access and secret key for the bucket. Data backup and restore in Secure Workload will not work with pre-authenticated links.

  Note	If you are using Cohesity as an object store, disable multi-part uploads while scheduling.

• HTTPS

  The data backup option supports only HTTPS interface with the object store. This is to ensure that data in transit to the object store is encrypted and secure. If the storage SSL/TSL certificate is signed by trusted third-party CA, the cluster will use them to authenticate the object store. In case the object store uses self-signed certificate, the public key or the CA can be uploaded by selecting the Use Server CA Certificate option.

• Server-side Encryption

  It is strongly recommended to turn ON server-side encryption for the bucket assigned to Secure Workload cluster. The cluster will use HTTPS to transfer data to object store. However, the object store should encrypt the objects to ensure that the data at rest is secure.

Note	If the Data Backup link under Platform is not available, contact Cisco Technical Assistance Center to enable the data backup and restore options. If the cluster is in standby mode, you will not be able to view the Data Backup link.

To configure data backup in Secure Workload, perform the following:

1. Planning: The data backup option provides a planner to test the access to the object store, determine the storage requirement, and the backup duration needed for each day. This can be used to experiment before configuring a schedule.

   To use data backup and restore calculators, navigate to Platform > Data Backup. If data backup and restore is not configured, this will navigate to the Data Backup landing page.

   

   To plan the data backup, use the following options:

   • Use Storage Planner

   • Use Capacity Planner

   Note	If you are unable to view the Data Backup option under Platform, ensure that you have the license to enable data backup and restore.

2. Configuring and scheduling data backup: Secure Workload will copy data to object store only in the configured time-window. While configuring backup for the first time, the pre-checks will run to ensure the FQDNs are resolvable and resolves to the right IP. After the initial validation, an update is pushed to registered software agents to switch to using FQDNs. Without FQDN, the agents cannot failover to another cluster after a disaster event. To support this, agents must be upgraded to the latest version supported by the cluster and all the agents should be able to resolve the sensor VIP FQDN. As of Secure Workload release 3.3 and later, only deep visibility and enforcement agents support data backup and restore and will switch to using FQDN.

   To create a schedule and configure data backup, see Configure Data Backup.

After the configuration of data backup, backup is triggered everyday at a scheduled time, unless continuous mode is enabled. Status of the backups can be seen on the Data Backup dashboard by navigating to Platform > Data Backup.

Time since last successful checkpoint should be less than 24 hours + the time it takes to checkpoint. For example, if the checkpoint + backup takes around 6 hours, then the time since last successful checkpoint should be less than 30 hours.

The following graphs provide additional information:

• Checkpoint Duration: This graph shows the trendline for the amount of time the checkpoint takes.

• Upload Duration: This graph shows the trendline for how long it takes to upload the checkpoint to the backup.

• Checkpoint Size: This graph shows the trendline for the size of the checkpoint.

• Upload Bandwidth: This graph shows the trendline for the upload bandwidth.

The table shows all the checkpoints. Checkpoint labels can be edited and the labels will be available while choosing a checkpoint to restore data on the standby cluster.

A checkpoint transitions through multiple states and these are the possible states:

• Created/Pending: Checkpoint is just created and waiting to be copied

• Running: Data is getting actively backed up to external storage

• Success: Checkpoint is complete and is successful; can be used for data restore

• Failed: Checkpoint is complete and has failed; cannot be used for data restore

• Deleting/Deleted: An aged-out checkpoint is being deleted or is deleted

To change the schedule or the bucket, click on Edit Schedule. To complete the wizard, see the Configure Data Backup section.

To troubleshoot any errors during the creation of checkpoints, see Troubleshooting: Data Backup and Restore.

Secure Workload cluster manages the lifecycle of objects in the bucket. You must not delete or add objects to the bucket. Doing so may lead to inconsistencies and corrupt successful checkpoints. In the configuration wizard, the maximum storage to be used must be specified. Secure Workload will ensure the usage of bucket will stay within the configured limit. There is a storage retention service that ages out objects and deletes them from the bucket. After the storage usage reaches a threshold (80% of the bucket capacity), computed based on the configured maximum storage and incoming data rate, the retention will try to delete un-preserved checkpoints to reduce the usage to below the threshold. The retention will also keep a minimum of two successful checkpoints at any time and all the preserved checkpoints, whichever is more. If retention cannot delete any checkpoints to make space, checkpoints will start failing.

As new checkpoints are created, old ones will age-out and are deleted. However, checkpoints can be preserved, preventing it from being deleted by retention. A preserved checkpoint will not be deleted. If there are multiple preserved checkpoints, at some point the storage will be insufficient for new objects and aged-out checkpoints cannot be deleted because they were preserved. As a best practice, preserve checkpoints on a need basis and update the Label for the checkpoint with the reason and validity as a reference. To preserve a checkpoint, click on the lock icon against the required checkpoint.

• To restore using backed up data, a cluster must be in the DBR standby mode. Currently, you can set a cluster to standby mode only during initial setup.

• After the cluster is in standby mode, choose Platform from the navigation pane to access the data restore option.

Secure Workload supports the following combinations:

This section describes the Recovery Time Objective (RTO) and Recovery Point Objective (RPO) for the data backup and restore solution.

A backup initiated on the primary cluster requires some time to complete depending on the amount of data being backed up and the backup configuration. The different modes of backup defines the RPO for the solution.

• If scheduled, non-continuous backup is used and the backup is initiated once in a day. If a disaster occurs then the maximum time of lost data will be approximately 24 hours, plus time taken to copy the data to the backup storage. Therefore, the RPO is at least 24 hours.

• If continuous mode backup is used then a new backup is initiated 15 minutes after the previous backup. Each backup consumes a certain amount of time to create and then a certain amount of time to upload the data to the backup storage. The first backup is a full backup and the subsequent backups are incremental backups, the incremental backups do not take much time. If a disaster occurs, the amount of the data lost will be the sum of the time taken to create the backup and the time taken to upload the backup to the storage. Typically the RPO in this case will be approximately a few minutes to an hour.

When restoring a cluster, mandatory data is first prefetched from the storage, then mandatory restore phase is triggered. The UI is not available during the mandatory restore phase. After the mandatory restore is complete, the UI is available for usage. The rest of the data is restored in the lazy restore phase. RTO in this case is the time taken until the UI is available for usage after mandatory phase is complete. RTO depends on the standby deployment mode.

• Cold Standby Mode: In this mode, the cluster must be deployed first which takes approximately a few hours. The cluster must then be configured with the backup storage credentials. Since this is the first time the backup is uploaded into the standby cluster, there will be a lot of mandatory data that needs to be retrieved and processed. The time for prefetch is approximately tens of minutes (depending on the quantity of data backed up). The mandatory restore phase takes approximately 30 minutes to complete. Together this forms the RTO time of approximately a few hours, primarily due to the time taken to boot and deploy the cluster.

• Luke Warm Standby Mode: In this mode, the cluster is already deployed but the backup storage is not configured. The cluster must be configured with the backup storage credentials. Since this is the first time the backup is uploaded into the standby cluster, there will be a lot of mandatory data that needs to be retrieved and processed. The time for prefetch is approximately tens of minutes (depending on the quantity of data backed up). The mandatory restore phase takes approximately 30 minutes to complete. Together this forms the RTO time of approximately an hour to two hours, depending on the amount of data backed up and time to pull the data from backup storage.

• Warm Standby Mode: In this mode, the cluster is already deployed, the backup storage is configured, and prefetch is retrieving data from the storage. The cluster can now be restored, which will trigger the mandatory restore phase, which takes approximately 30 minutes to complete. This forms the RTO time of approximately 30 minutes. Note that there is some delay from when the backup is uploaded from the active to the storage to the time the backup is pulled by the standby. This is approximately a few minutes. If the latest backup from the active (prior to it experiencing a disaster event) has not been prefetched to the standby, you must wait for a few minutes for it to be retrieved.

When data backup and restore is enabled on the cluster, it is recommended to deactivate the schedule before starting the upgrade. See Deactivating Backup Schedule. This ensures that a successful backup exists before upgrade is started and that no new backup is being uploaded. A schedule must be deactivated when a checkpoint is not in progress, to avoid creating a failed checkpoint.

The key components of a Secure Workload cluster are:

• Bare metal servers that host multiple VMs, which in turn, host many services.

• Cisco UCS C-Series Rack Servers with Cisco Nexus 9300 Series switches that contribute to an integrated high-performance network.

• Hardware-based appliance models in either a small or large form factor to support a specific number of workloads:

  • Small form factor deployment with six servers and two Cisco Nexus 9300 switches.

  • Large form factor deployment with 36 servers and three Cisco Nexus 9300 switches.

• There is no impact to the cluster operation at any point in time.

• There is no single point of failure. If any of the nodes or VMs within a cluster fail, it does not result in the failure of the entire cluster.

• There is minimal downtime in recovery from failure because of services, nodes, or VMs.

• There is no impact on the connections that are maintained by software agents to a Secure Workload cluster. The agents communicate with all the available collectors in the cluster. If a collector or VM fails, the software agents’ connections to the other instances of the collectors ensure that the flow of data is not interrupted and there is no loss of functionality.

• The cluster services communicate with external orchestrators. When the primary instance of a service fails, the secondary instances take over to ensure the communication with external orchestrators is not lost.

Services Failure

When a service fails on a node, another instance of that particular service picks up the functions of the failed service and continues to run.

Table 11. Services Failure Impact and Recovery

Impact	No visible impact.
Recovery	Minimal downtime for the UI or dependent services to continue to run from the secondary instances. Recovery is automatic.

VM Failure

When one of the VMs fails, secondary VMs are available. The services on the secondary VMs pick up the services that the failed VM was running. Meanwhile, Secure Workload restarts the failed VM to recover it. For example, as illustrated in the Figure: Failure Scenario of a VM, when a VM, in this instance, VM1, fails, the services running on it also fails. The secondary VMs continue to be operational and the secondary instances pick up the services that the failed VM was running.

For services provided by symmetric VMs, such as collectordatamovers, datanode, nodemanager, and druidHistoricalBroker VMs, multiple VMs can fail but the applications will continue to function at reduced capacity.

Table 12. Symmetric VM Types

Service Type	Total VMs	Number of VM Failures Supported
Datanode	6	4
DruidHistorical	4	2
CollectorDataMover	6	5
NodeManager	6	4
UI/ AppServer	2	1

Note	The nonsymmetric VM types tolerate only one VM failure before the corresponding services are rendered unavailable.

Table 13. VM Failure Impact and Recovery

Impact	No visible impact.
Recovery	Minimal downtime for the UI or dependent services to continue to run from the secondary instances on other VMs. Recovery is automatic. However, if a VM remains inactive, contact Cisco Technical Assistance Center to troubleshoot the issue. In a few instances, you may have to replace the bare metal.

Node Failure

Table 14. Number of Node Failures Tolerated

Node Failures	8 RU	39 RU
Number of node failures that are tolerated for high availability	1	1*

* In 39 RU clusters, single node failure is always tolerated. A second node failure might be allowed as long as the two failed nodes do not host VMs for a 2 VM or 3 VM service, such as orchestrators, Redis, MongoDB, Elasticsearch, enforcementpolicystore, AppServer, ZooKeeper, TSDB, Grafana, and so on. In general, the second node failure results in a critical service becoming unavailable because of two VMs being affected.

Caution

We recommend that you immediately restore the failed node because the failure of a second node will most likely result in an outage.

Table 15. Node Failure Impact and Recovery

Impact	No impact in the functionality of the cluster. However, contact Cisco Technical Assistance Center to replace the failed node immediately. Failure of a second node will most likely result in an outage.
Recovery	Minimal downtime. If a node fails, we recommend that you contact Cisco Technical Assistance Center for assistance to remove the faulty node and replace it with another node.

Network Switch Failure

The switches in Secure Workload always remain active. In the 8RU form-factor deployment, there is no impact if a switch fails. In the 39RU form-factor deployment, the clusters experience half the input capacity if a switch fails.

Note	The switches in the Secure Workload cluster do not have the recommended port density to support the VPC configuration for public networks.

Table 16. Number of Switch Failures Tolerated

Form Factor	8 RU	39 RU
Number of switch failures that are tolerated for high availability	1 Note   If two or more switches fail, it is likely to have an impact on the entire functionality of the cluster.	1 Note   A single switch failure results in half input capacity. Two or more failures are likely to impact the entire functionality of the cluster.

Table 17. Network Switch Failure Impact and Recovery

Impact

A faulty switch or network card on a bare metal causes loss of network connectivity within the cluster. There is no impact in the functionality of a cluster because of a single switch failure. However, two or more failures are likely to impact the entire functionality of the cluster. Connectivity issues to multiple VMs on a cluster, or intermittent and prolonged connectivity problems result in unpredictable behaviour within the cluster.

Recovery

Recovery is automatic. Contact Cisco Technical Assistance Center for assistance with faulty switches or network cards on bare metals.

Supported upgrade types for a Secure Workload cluster:

• Full Upgrade: To initiate full upgrade, from the navigation pane, choose Platform > Upgrade/Reboot/Shutdown. In the Upgrade tab, select Upgrade. During the full upgrade process, the VMs are powered off, and VMs are upgraded and redeployed. There is a cluster downtime during which the Secure Workload UI is inaccessible.

• Patch Upgrade: Patch upgrade minimizes the cluster downtime. The services that must be patched are updated and do not result in VM restarts. The downtime is usually in the order of a few minutes. To initiate patch upgrade, select Patch Upgrade and click Send Patch Upgrade Link.

An email with a link is sent to the registered email address to initiate the upgrade.

Before sending the email, the orchestrator runs several verification checks to make sure the cluster is upgradable. The checks include:

• Checks to see there are no decommissioned nodes.

• Checks each bare metal to make sure there are no hardware failures, covering the following:

  • Drive failure

  • Drive predicted failure.

  • Drive missing

  • StorCLI failures

  • MCE log failures

• Checks to ensure that bare metals are in the commissioned state, nothing fewer than 36 servers for 39RU and six for 8RU.

Note

If there are any failures, an upgrade link is not sent to the registered email address and a 500 error is displayed with information such as HW failure, or missing host and check orchestrator logs for more information. In this scenario, use explore to tail -100 on /local/logs/tetration/orchestrator/orchestrator.log in the host orchestrator.service.consul. The log provides detailed information about which of the three checks caused the failure. It usually requires fixing the hardware and recommissioning the node. Restart the upgrade process.

After you click the upgrade link that is received on your email, the Secure Workload Setup page is displayed. The Setup UI is used to deploy or upgrade the cluster. The landing page displays the list of RPMs that are currently deployed in the cluster. You can upload the RPMs to upgrade the cluster.

Upload the RPMs in the order that is shown on setup UI. Not uploading RPMs in the correct order result in upload failures and the Continue button remains disabled. The order is:

1. tetration_os_rpminstall_k9

2. tetration_os_UcsFirmware_k9

3. tetration_os_adhoc_k9

4. tetration_os_mother_rpm_k9

5. tetration_os_enforcement_k9

6. tetration_os_base_rpm_k9

Note	For Secure Workload Virtual clusters deployed on vSphere, ensure that you also upgrade the tetration_os_ova_k9 RPM and that you do not upload the tetration_os_base_rpm_k9 RPM.

To view the logs for each upload, click the Log symbol on the left of every RPM. Also, uploads that failed will be marked RED in color.

For detailed instructions, see the Cisco Secure Workload Upgrade Guide.

The next step in upgrading the cluster is to update the site information. Not all site information fields are updateable. Only the following fields can be updated:

• SSH public Key

• Sentinel Alert Email (for Bosun)

• CIMC Internal Network

• CIMC Internal Network Gateway

• External Network

  Note	Do not change the existing external network, you can add additional networks by appending to the existing ones. Changing or Removing existing network will make the cluster unusable.

• DNS Resolvers

• DNS Domain

• NTP Servers

• SMTP Server

• SMTP Port

• SMTP Username (Optional)

• SMTP Password (Optional)

• Syslog Server (Optional)

• Syslog Port (Optional)

• Syslog Severity (Optional)

Note

The syslog server severity ranges from critical to informational. Severity needs to be set to warning or higher (informational) for bosun alerts. From 3.1 version, External syslog via setup UI is not supported. Configure TAN Appliance to export data to syslog. For more details, see External syslog tunneling moving to TAN. Secure Workload supports secure SMTP communication with mail servers that support SSL or TLS communication using the STARTTLS command. The standard port for servers that support secure traffic is usually 587/TCP, but many servers also accept secure communication on the standard 25/TCP port. Secure Workload does not support the SMTPS protocol for communicating with external mail servers.

The rest of the fields are not updatable. If there are no changes, click Continue to trigger the Pre-Upgrade Checks, else update the fields and then click Continue.

Before upgrading the cluster, a few checks are performed on the cluster to ensure things are in order. The following preupgrade checks are performed:

• RPM version checks: Checks to ensure all the RPMs are uploaded and the version is correct. It does not check if the order was correct, just checks if it was uploaded. Note that order checks are done as a part of the upload itself.

• Site Linter: Performs Site Info linting

• Switch Config: Configures the Leafs or Spine switches

• Site Checker: Performs DNS, NTP, and SMTP server checks. Sends an email with a token, the email is sent to the primary site admin account. If any of the services - DNS, NTP or SMTP is not configured, this step fails.

• Token Validation: Enter the token that is sent in the email and continue the upgrade process.

Occasionally, there may be hardware failures or the cluster may not be ready to be upgraded after scheduling an upgrade and while initiating an upgrade. These errors must be fixed before proceeding with upgrades. Instead of waiting for an upgrade window, you can initiate preupgrade checks which can be run any number of times and anytime, except when upgrade, patch upgrade, or reboot is initiated.

To run preupgrade checks:

1. In the Upgrade tab, click Start Upgrade Precheck.

   This initiates the preupgrade checks and is transitioned to a running state.

   

2. After all the checks that are run by orchestrators pass, an email with a token is sent to the registered email ID. Enter the token to complete the preupgrade checks.

   

You can verify the status of the checks. If there are any failures during preupgrade checks, you can view the failed checks and the respective check transition to the failed state.

If DBR is enabled on the cluster, also see Upgrades (with DBR).

Users with Customer Support role can access the snapshot tool by selecting Troubleshoot > Snapshots from the navigation bar at the left side of the window.

The Snapshot tool can be used to create a Classic Snapshot or a Cisco Integrated Management Controller (CIMC) technical support bundles. Clicking on the Create Snapshot button on the Snapshot file list page loads a page to choose a Classic Snapshot or a CIMC Snapshot (technical support bundle). The option to choose a CIMC Snapshot is disabled on Secure Workload Software Only (ESXi) and Secure Workload SaaS.

Clicking on the Classic Snapshot button loads the Snapshot tool runner user interface:

Clicking on the CIMC Snapshot button loads the CIMC Technical Support tool runner user interface:

Select Create Snapshot with the default options, the Snapshot tool collects:

• Logs

• State of Hadoop or YARN application and logs

• Alert history

• Numerous TSDB statistics

It is possible to override the defaults and specify certain options.

• logs options

  • max log days - number of days of logs to collect, default 2.

  • max log size - maximum number of bytes per log to collect, default 128kb.

  • hosts - hosts to get logs/status from, default all.

  • logfiles - regex of logs to be fetched, default all.

• yarn options

  • yarn app state - application states (RUNNING, FAILED, KILLED, UNASSIGNED, etc) to get information for, default all.

• alerts options

  • alert days - the number of days worth of alert data to collect.

• tsdb options

  • tsdb days - the number of days worth of tsdb data to collect, increasing this can create very large Snapshots.

• fulltsdb options

  • fulltsdb - a JSON object that can be used to specify startTime, endTime fullDumpPath, localDumpFile and nameFilterIncludeRegex to limit which metrics are collected.

• comments - can be added to describe why or who is collecting the snapshot.

After selecting Create Snapshot, a progress bar for the snapshot is displayed at the top of the Snapshot file list page. When the snapshot completes, it can be downloaded using the Download button on the Snapshots file list page. Only one snapshot can be collected at a time.

On the CIMC Snapshot (technical support bundle) page, select the serial number of the node the CIMC Technical Support Bundle should be created for and click the Create Snapshot button. A progress bar for the CIMC Technical Support Bundle collection is displayed in the Snapshot file list page and the comments section will reflect that the CIMC Technical Support Bundle collection has been triggered. When the CIMC Technical Support Bundle collection is complete, the file can be downloaded from the Snapshot file list page.

Untarring a snapshot creates a ./clustername_snapshot directory that contains the logs for each machine. The logs are saved as text files that contain the data from several directories from the machines. The Snapshot also saves all the Hadoop/TSDB data that was captured in JSON format.

When opening the packaged index.html in a browser, there are tabs for:

• Terse list of alert state changes.

  

• Reproduction of grafana dashboards.

  

• Reproduction of the Hadoop Resource Manager front end that contains jobs and their state. Selecting a job displays the logs for the job.

  

• List of all logs collected.

  

The snapshot service can be used to run service commands, but it requires Customer Support privileges.

Using the Explore tool (Troubleshoot > Maintenance Explorer), you can hit arbitrary URIs within the cluster:

The Explore tool only appears for users with Customer Support privileges.

The snapshot service runs on port 15151 of every node. It listens only on the internal network (not exposed externally) and has POST endpoints for various commands.

The URI you must hit is POST http://<hostname>:15151/<cmd>?args=<args>, where args are space separated and URI encoded. It does not run your command with a shell. This would avoid allowing anything to be run.

Endpoints of a snapshot are defined for:

• snapshot 0.2.5

  ̶Is

  ̶ svstatus, svrestart - runs sv status, sv restart Example:1.1.11.15:15151/svrestart?args=snapshot

  ̶ hadoopls runs hadoop fs -ls <args>

  ̶ hadoopdu - runs hadoop fs -du <args>

  ̶ ps Example: 1.1.11.31:15151/ps?args=eafux

  ̶ du

  ̶ ambari - runs ambari_service.py

  ̶ monit

  ̶ MegaCli64 (/usr/bin/MegaCli64)

  ̶ service

  ̶ hadoopfsck - runs hadoop -fsck

• snapshot 0.2.6

  ̶ makecurrent - runs make -C /local/deploy-ansible current

  ̶ netstat

• snapshot 0.2.7 (run as uid “nobody”)

  ̶ cat

  ̶ head

  ̶ tail

  ̶ grep

  ̶ ip -6 neighbor

  ̶ ip address

  ̶ ip neighbor

There is another endpoint, POST /runsigned, which will run shell scripts that are signed by Secure Workload. It runs gpg -d on the POSTed data. If it can be verified against a signature, it runs the encrypted text under a shell. This means importing a public key on each server as part of the Ansible setup and the need to keep the private key secure.

Users with Customer Support privileges can use Run Book by selecting Troubleshoot > Maintenance Explorer from the navigation bar at the left side of the window. Select POST from the drop-down menu. (Otherwise you will receive Page Not Found errors when running commands.)

Using the snapshot REST endpoint to restart services:

• druid: 1.1.11.17:15151/service?args=supervisord%20restart

  ̶ druid hosts are all IPs .17 through .24; .17, .18 are coordinators, .19 is the indexer, and .20-.24 are brokers

• hadoop pipeline launchers:

  ̶ 1.1.11.25:15151/svrestart?args=activeflowpipeline

  ̶ 1.1.11.25:15151/svrestart?args=adm

  ̶ 1.1.11.25:15151/svrestart?args=batchmover_bidir

  ̶ 1.1.11.25:15151/svrestart?args=batchmover_machineinfo

  ̶ 1.1.11.25:15151/svrestart?args=BDPipeline

  ̶ 1.1.11.25:15151/svrestart?args=mongo_indexer

  ̶ 1.1.11.25:15151/svrestart?args=retentionPipeline

• policy engine

  ̶ 1.1.11.25:15151/svrestart?args=policy_server

• wss

  ̶ 1.1.11.47:15151/svrestart?args=wss

If a hardware failure is detected upon restart of a cluster after power shutdown, currently the cluster gets stuck in a state where we can neither run Reboot workflow to get services stable nor run Commission workflow as down services result in commissioning failure. This feature is expected to help in such scenarios by allowing the user to reboot (upgrade) with a bad hardware, after which the regular RMA process for the failed bare metal can be performed.

The user is expected to use a post to explore the endpoint with serial of the bare metal to be excluded:

1. Action: POST

2. Host: orchestrator.service.consul

3. Endpoint: exclude_bms?method=POST

4. Body: {“baremetal”: [“BMSERIAL”]}

The orchestrator performs a few checks to determine if the exclusion is feasible. In which case, it will setup a few consul keys and return a success message indicating which bare metal and VMs will be excluded in the next reboot/upgrade workflow. If the bare metals include certain VMs, they cannot be excluded as described in the Limitation section below, the explore endpoint replies back with the message indicating why the exclusion is not possible. After successful post on the explore endpoint, the user can initiate reboot/upgrade through the main GUI and proceed with reboot as usual. At the end of the upgrade, we remove the exclude bm list. If there is a need to run upgrade or reboot again with exclude BMs, users are expected to post to the bmexclude explore endpoint again.

Limitations

The following VMs cannot be excluded:

• namenode

• secondaryNamenode

• mongodb

• mongodbArbiter

After the prechecks passes, you can proceed to decommission the disk. The progress of decommission will be shown on the disk replacement wizard. When the progress of decommission reaches 100%, all the decommissioned disks' status changes to UNUSED.

After decommissioning the disks, remove the disks and replace with new disks. To help with this process, we have added disk and server locator LED access on the replace page. Ensure you switch off the server and disks locator LEDs.

Disks can be physically replaced in any order but they must be reconfigured in smallest to largest slot numbers for a given server. This order is enforced through on the UI and the backend. On the UI, you will have a replace button active for disk with the lowest slot number with the status UNUSED.

When all the disks are replaced, proceed to commission. Similar to decommission, we need to run a set of prechecks before we can continue to commission.

Progress of commission is monitored on the disk commission page. At the end successful commission, the status of all disks change to HEALTHY.

After you deploy the VMs and there is a failure, you can recover using the Resume Commission button. To continue commission of disk, click the Resume Commission button to restart the post-deploy playbooks.

In case of any failure before you deploy the VMs, the disks that were commissioned earlier will have their status changed to UNHEALTHY. That will require us to restart the replacement process from the decommission of UNHEALTHY disks.

In case of any other disks than the ones that are being replaced fails while disk commission is in progress, notice of this failure will be displayed on the disk replacement wizard after the ongoing commission process finishes, either in success or failure.

In cases of resumable failures, user will have two options in what next steps to take.

1. They can try to resume and complete the current commission and perform the disk replacement process for the new failures later.

2. Alternatively, they can start decommission of newly failed disk and perform commission of all the disks together.

This second path is the only path available in cases of nonresumable failures. If the post-deploy failure is caused due to the newly failed disks, the second path will again be only way forward, although we have resume button available.

• Disk containing server root volumes cannot be replaced using this procedure. Such disk failures must be corrected using server maintainence process.

• Disk commissioning can happen only when all servers are active and in commissioned state. See Special Handling section that describes how to proceed in the cases where a combination of disk and server replacement is needed.

• SSD disks are too expensive and have a very low failure rate and therefore we do not want to lose valuable capacity for redundant data storage.

• On M6 clusters originally deployed with 3.8 software, when a server is commissioned with 3.9 software, RAID configuration will be applied on the HDD drives. This will cause a cluster to contain some nodes with RAID and some with the non-RAID disk configuration from 3.8. It is likely that your Secure Workload 39RU hardware originally shipped with 3.9 already installed, but some early M6s were shipped with 3.8 deployed.

• You can convert a cluster to RAID if server decommission and commission is performed progressively across all servers after upgrading to 3.9 software.

• M6 8RU clusters are all-SSD nodes and RAID is not configured on SSD drives and therefore 8RUs do not get RAID.

• Drive configuration on older generations (M4/M5) prevents us from supporting RAID on those generations of Secure Workload hardware.

Shutting down the cluster stops all running Secure Workload processes, and powers down all individual nodes. Perform the following steps to shut down the cluster.

To recover the cluster after shutdown, power on the bare metals. When all the individual bare metals are up, the UI becomes accessible. After logging into the cluster, reboot the cluster to render the cluster operational.

Note	You must reboot the cluster after a shutdown to render it operational.

To view the previously run cluster maintenance jobs:

1. Navigate to Platform > Upgrade/Reboot/Shutdown, and then click the History tab.

   The cluster operation column lists the cluster tasks such as deploy, upgrade, reboot, or shutdown.

2. To download logs of the cluster jobs, click Download Logs.