Monitoring Cisco VTS

The following sections provide details about Monit and collectd features that enable you to monitor Cisco VTS.

Note
Monitoring features (collectd and Monit) are not supported for Data Plane (VTF) when VTF is in VTEP mode, on vCenter (VM mode).

Monit is a process monitoring tool. It collects and displays metrics related to memory consumption, CPU usage, swap information processes, file system, in a dashboard. For each process in each component, the dashboard shows status, uptime, CPU unitization, Memory, and Read/Write bytes on the disk. It is packaged as part of VTC, VTSR, and VTF, and will get installed as part of the respective installations. See Monitoring Cisco VTS Infrastructure using Monit for details.

collectd is a system statistics daemon which collects system and application performance metrics periodically and provides mechanisms to store the values. See https://collectd.org/for details about collectd. Cisco VTS installation installs collectd. On a new Cisco VTS installation, collectD plugins are preconfigured to load and run. These plugins have their configurations already saved in VTC. collectd collects various statistics related to VTC and VTF, which includes CPU, memory, number tenants, networks routers etc, based on the plugins that you enable. The metrics can be sent to an external location you specify, in JSON format, which can be used for further processing. See Monitoring Cisco VTS Infrastructure using collectd for details about how collectd is used in Cisco VTS.

Cisco VTS displays the Policy Plance monitoring details in the Monitor page, and the Data Plane and Control Plane monitoring detials in the Monitor Sitepage.

Monitoring Cisco VTS Infrastructure using Monit

Monit is used to collect status of all the services that are running on the VTS and VTSR VMs as well as compute nodes running VTF. The Policy Plane monitoring details are displayed in a dashboard under Monitor. The Data Plane and Control Plane monitoring details are displayed underMonitor Site.

The intervals when Monit will collect metrics for each of the Cisco VTS components are:

  • VTC—60 seconds

  • VTSR—As entered in the vtsr_template.cfg file while installing VTSR.

  • VTF—30 seconds


Note

For VTC and VTF, the intervals are fixed and you cannot change these values.

This means that, when an event happens, the Cisco VTS UI will show the appropriate status only after this interval has passed.

About Monit Username and Passwords

  • For Policy Plane (VTC)—Monit is packaged as part of VTC. Monit, when installed, will have a default username and password. The default monit credentials for VTC when installed are:
    • Username—monit-ro

    • Password—monit-ro

    Upon logging into Cisco VTS for the first time after installation, the admin needs to enter this default username and password in the VTC GUI at Global Settings > Monitoring Settings > Monit Settings to view the VTC monitoring information in the Monit Dashboard.
  • For Control Plane (VTSR)—Monit is packaged as part of VTSR.
    The default credentials for VTSR is entered while installing VTSR, by modifying the below two properties in vtsr_template.cfg: 
    #VTSR_OPER_USERNAME="monit-ro-oper"
# Password needs an encrypted value
# Example : "openssl passwd -1 -salt <salt-string> <password>"
#VTSR_OPER_PASSWORD="$1$cisco$b88M8bkCN2ZpXgEEc2sG9/"

    Note

    See the Installing VTSR section in the Cisco VTS 2.6.3 Installation Guide, for details.

    Upon logging into Cisco VTS for the first time after installation, the admin needs to enter this default username and password in the VTC GUI at Administration > Site Monitoring Settings > Monit Settings to view the VTSR monitoring information in the Monit Dashboard.
  • For Data Plane (VTF)—Monit is packaged as part of VTF.

    The default monit credentials for VTFs are:

    • Username—monit-ro

    • Password—monit-ro

    Upon logging into Cisco VTS for the first time after installation, the admin needs to enter this default username and password in the VTC GUI at Administration > Site Monitoring Settings > Monit Settings to view the VTF monitoring information in the Monit Dashboard.

  • For Data Plane (VTF) deployed via OpenStack Platform Director (OSPD)—When VTF is installed via OSPD, the Monit related properties need to be updated in the neutron-cisco-vts.yaml file. See the Installing Cisco VTS 2.6.3 Components in OpenStack using Red Hat Enterprise Linux OpenStack Directodocument for details.

Changing Monit Password Subsequently

To change Monit password for VTC, VTF, or VTSR, the admin needs to run the following script: 
/opt/vts/bin/update_monit_credentials.sh.

  • To change password for Policy Plane, run:

    /opt/vts/bin/update_monit_credentials.sh policy-plane <monit-username> <monit-password>

  • To change password for Data Plane, run:

    /opt/vts/bin/update_monit_credentials.sh data-plane <monit-username> <monit-password> <site id>

  • To change password for Control Plane, run:

    /opt/vts/bin/update_monit_credentials.sh control-plane <monit-username> <monit-password> <site id>

For VTC, the username cannot be changed. The default is monit-ro.

In HA mode, the password has to be changed on the Master.

To change Monit password for VTF deployed via OSPD, update the following properties under Monit-Configuration section in the neutron-cisco-vts.yaml file. See the Installing Cisco VTS 2.6.3 Components in OpenStack using Red Hat Enterprise Linux OpenStack Directordocument for details.

Metrics Collected using Monit-D

Monit runs in VTC Master, VTC Slave, VTSR Master, VTSR slave, and all the VTFs. The VTS UI Monitoring page displays the monitoring status, on-demand.

Following are the intervals when Monit will collect metrics for each of the planes. This means that, when an event happens, the Cisco VTS UI displays the appropriate status only after this interval has passed.
  • VTC—60 seconds.

  • VTSR—Based on what is configured in the template.cfg file during installation.

  • VTF—30 seconds.

Metrics Collected for VTC

The following metrics are collected:

  • Process: The following VTS processes can be monitored.

    • Corosync

    • Pacemaker

    • Filebeat

    • Logstash

    • Collectd

    • Monit

    • Nso

    • Ntpd

    • Sshd

    • Solr

    • Nginx

    • Nodejs

    • Tomcat

    • Vtsweb

  • File System

    • Root

    • Boot

  • Network

    • Management

    • Underlay

Metrics Collected for VTSR

The following metrics are collected:
  • Process—The following VTSR processes can be monitored.

    • Confd

    • Rc

    • Dl

    • cfg_dl

    • redis

    • stunnel

    • pacemaker

    • corosync

    • logstash

    • monit

    • filebrat-god

    • filebeat

Setting up Monit Credentials

You must set up the credentials for the Policy Plane (VTC), Control Plane (VTSR), and Data Plane (VTF), to enable you to access the metrics collected by Monit via the Cisco VTS UI. See Monitoring Cisco VTS Infrastructure using Monit for details about Monit credentials for VTC, VTSR, and VTF.

Setting up Policy Plane Credentials

To set up the credentials to enable accessing Policy Plane metrics:

Procedure
Step 1

Go to Global Settings > Monitoring Settings.

Step 2

Click the Policy Plane tab.

Step 3

Enter the credentials.

Note 
The username is monit-ro by default, and cannot be changed. The Password should match the one that was configured during Monit setup.

Setting up Control Plane Credentials

To set up the credentials to enable accessing Control Plane metrics:

Procedure
Step 1

Go to Administration > Site Monitoring Settings.

Step 2

Click the Control Plane tab.

Step 3

Enter the Username and Password that was set during Monit setup.

Setting up Data Plane Credentials

To set up the credentials to enable accessing Data Plane metrics:

Procedure
Step 1

Go to Administration > Site Monitoring Settings.

Step 2

Click the Data Plane tab.

Step 3

Enter the Username and Password that was set during Monit setup.

Viewing Metrics Collected by Monit

The Policy Plane metrics collected by Monit is displayed in Cisco VTS > Monitor UI. The Data Plane and Conrol Plane metrcis are displayed in Cisco VTS > Monitor Sites UI.

Viewing Policy Plane Metrics

To view the metrics collected for Policy Plane (VTC):

Procedure
Step 1

Go to Monitor.

Step 2

Click the Policy Plane tab.

Note 
Monitoring information is displayed only after you complete the Monitoring Settings under Global Settings.

Viewing Control Plane Metrics

To view the metrics collected for Control Plane (VTSR):

Procedure
Step 1

Go to Monitor.

Step 2

Click the Control Plane tab.

Note 
Monitoring information is displayed only after you complete the Site Monitoring Settings under Administration.

Viewing Data Plane Metrics

To view the metrics collected for Data Plane (VTF):

Procedure
Step 1

Go to Monitor.

Step 2

Click the Data Plane tab.

Note 
Monitoring information is displayed only after you complete the Site Monitoring Settings under Administration.
Step 3

Select the VTF IP address from the drop-down list.

Note 
Monitoring information is displayed only after you complete the Site Monitoring Settings under Administration. In an HA set up, you must specify the static route to reach the VTF on both Master and Slave. This is to ensure that VTF statistics is displayed even when a Master VTC is switched over to Slave VTC.

Monitoring Cisco VTS Infrastructure using collectd

Cisco VTS embeds collectd to collect metrics and statistics of VTS components. Currently, collectd is embedded as part of VTC and VTFs.

collectd is a system statistics daemon which collects system and application performance metrics periodically and provides mechanisms to store the values.

collectd starts running upon Cisco VTS installation and is configured for a default collection interval of 120 seconds. In an HA setup, collectd runs on both master and slave VTCs. At the configured interval, it will invoke the Input plugins.

The write_log output plugin logs the metrics in the local VTC at /opt/vts/log/collectd/metrics.log, and write_http plugin (when configured) pushes the metrics to an external location.

List of Plugins for VTC

  1. Py Custom Plugin—Gets periodically called by collectd; calls the UI backend API to get the stats in JSON format.

  2. CPU—Inbuilt plugin in collectd

  3. Memory—Inbuilt plugin in collectd

  4. Load—Inbuilt plugin in collectd

  5. Interface—Inbuilt plugin in collectd

  6. Disk—Inbuilt plugin in collectd
  7. log_file; write_log—Used to log the metrics locally.
  8. write_http—You must configure this plugin if you want to forward the metrics in JSON format to the centralized collectd server.

Note

Except write_http, all plugins are available by default upon installation.

Plugin Configuration for VTC and VTF

collectd can be configured for Policy Plane (VTC) from the Global Settings > Monitoring Settings > Collectd Settings page. collectd configuration for Data Plane can be done at Administration > Site Monitoring Settings.

Any change to the Policy Plane collectd settings and plugins would take a maximum of three minutes to get reflected in the VTC collectd process. Any change to the Data Plane Collectd settings and plugins is done immediately but would depend on the number of VTFs to get updated.

Following collectd parameters and plugin details can be set up:
  1. collectd Interval—This is interval for collect-d to collect the metrics. This is per plane. The default is 120 seconds.
  2. Enable/Disable collectd—This toggle switch will help you to enable/disable collectd for all the collectd plugins within that Plane. For VTC, in an HA setup, it would disable/enable collectd process for both master and slave. For VTFs, it would disable/enable collectd process for all the VTFs.
  3. Plugin Configuration—You can configure any collectd plugin as required.
    1. Plugin Name—Name of the plugin.

      See collectd Plugin Configuration for the default plugin configurations Cisco VTS supports.

      Following are the plugins that are supported:
      • CPU
      • Python
      • log_logstash
      • write_log
      • logfile—The log file location: /opt/vts/log/collectd/metrics.log.

        Note

        For VTF, you have to configure the log_file plugin. This is required to write the output to a specific log file. Otherwise it is sent to /var/log/messages.

      • Interface
      • Memory
      • Load
      • write_http—You must configure this plugin if you want to forward the metrics in JSON format to the centralized collectd server.
    2. Plugin Config—In xml format. 
      LoadPlugin {plugin-name}
        <Plugin {plugin-name}>
          {parameters of the  plugin}
        </Plugin>

      Sample write_http plugin config: 

      <LoadPlugin write_http>
     FlushInterval 10
</LoadPlugin>
<Plugin write_http>
   <Node "example">
      URL "http://10.10.10.10/centralized-collectd"
      Format "JSON"
      BufferSize 10240
   </Node>
</Plugin>

      FlushInterval ensures that the payloads are sent to the external server at predefined intervals. If the FlushInterval is lesser than collectd-interval, then collectd interval would take precedence, since there is nothing to send when the read-interval has not lapsed.

      If the FlushInterval is greater than collect-interval, then FlushInterval will take precedence and payloads will be sent as per the FlushInterval, provided the buffer does not get full.


      Note

      For the FlushInterval to be honored, we recommend that you keep a bigger buffersize (40960). The buffersize can depend on the statistics collected, which depends on scale.

    3. Enable/Disable Plugin—You can enable or disable the plugins via the Cisco VTS UI.

Plugin Configuration for VTF Deployed via OSPD

For changing the VTF collectd plugin configuration while deploying via OSPD, you need to modify it in the neutron-cisco-vts.yaml file. Any change or addition to the plugins would need a change in this yaml file (under the Collectd Agent Configuration section). See the Installing Cisco VTS 2.6.1 Components in OpenStack using Red Hat Enterprise Linux OpenStack Director doc for details.

Metrics Collected by collectd

  • Default Metrics:

    • CPU

    • Memory

    • Load

    • Disk

    • Interface

    • Python

    • write_log

  • VTC Statistics:

    • Total Number of Tenants

    • Total Number of Tenants per VTEP (HW and SW)

    • Total Number of Networks

    • Total Number of Networks per Tenant

    • Total Number of Networks per VTEP (HW and SW)

    • Total Number of Routers

    • Total Number of Routers per Tenant

    • Total Number of Router per VTEP (HW and SW)

    • Total Number of Baremetal per VTEP

    • Total Number of Shared Networks

    • Total Number of Hosts/Servers

    • Total Number of H/W Vteps

    • Total Number of S/W Vteps (VTFs)

Metrics Sent

The following information from the VTS is sent:

Note

See collectd Output JSON Examples for output file examples.

  1. Master or Slave

  2. IP or Hostname of the VTC

  3. Stats Category—For example, number of tenants

  4. Stats Sub-Category—For example. the VTEP name if we have tenants per VTEP

  5. Count—Count of tenants

Setting up collectd Plugins

The Global Settings > Monitoring Settings > Collectd Settings page displays the Plugin Types and also shows whether the plugin is enabled or disabled, for the Policy Plane. The Administration > Site Monitoring Settings > Collectd Settings page displays the Plugin Types and also shows whether the plugin is enabled or disabled, for the Data Plane.

The Manage Settings and Plugin option allows you to manage the collectd settings and add/remove, enable/disable, and edit plugins, for both Policy Plane and Data Plane. See Monitoring Cisco VTS Infrastructure using collectd for details about usage of collectd in Cisco VTS.

Setting up Policy Plane Plugins

To set up collectd parameters for Policy Plane:

Procedure
Step 1

Go to Global Settings > Monitoring Settings > Collectd Settings.

Step 2

Click the Policy Plane tab.

Step 3

Click Manage Settings and Plugin.

You can specify the following:

  • Collection Interval—The collection interval for metrics collection. By default, this is 120 seconds. This can be between 10 and 1800 seconds.

  • Enable Collect D—Use the toggle switch to enable of disable collecd metrics collection. By default, this is set to Yes.

  • Plugin Type—Choose from the list of plugins packaged with the collectd server.

  • Plugin Config—Enter or edit the configuration for the plugin type you selected.

  • Enable Plugin—Use the toggle switch to enable of disable the selected plugin. By default, the selected plugin is enabled.

Use the + button to add plugins. Use the - button to remove the plugin.

Step 4

Click Save.

Setting up Data Plane Plugins

To set up collectd parameters for Data Plane:

Procedure
Step 1

Go to Administration > Site Monitoring Settings > Collectd Settings.

Step 2

Click the Control Plane tab.

Step 3

Click Manage Settings and Plugin.

You can specify the following:

  • Collection Interval—The collection interval for metrics collection. By default, this is 120 seconds. This can be between 10 and 1800 seconds.

  • Enable Collect D—Use the toggle switch to enable of disable collecd metrics collection. By default, this is set to Yes.

  • Plugin Type—Choose from the list of plugins packaged with the collectd server.

  • Plugin Config—Enter or edit the configuration for the plugin type you selected.

  • Enable Plugin—Use the toggle switch to enable of disable the selected plugin. By default, the selected plugin is enabled.

Use the + button to add plugins. Use the - button to remove the plugin.

Step 4

Click Save.